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PREFACE 


This document contains the proceedings of the training workshop on Advanced 
Learning Technologies and Learning Networks and their impact on Future Aerospace 
Workforce. The workshop was held at the Peninsula Workforce Development Center, 
Hampton, Virginia, April 2 - 3, 2003. The workshop was jointly sponsored by Old 
Dominion University and NASA. Workshop attendees came from NASA, other 
government agencies, industry, and universities. The objectives of the workshop were to: 
1) provide broad overviews of the diverse activities related to advanced learning 
technologies and learning environments, and 2) identify future directions for research that 
have high potential for aerospace workforce development. Eighteen half-hour overview- 
type presentations were made at the workshop. 
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INTRODUCTION 


The engineering profession resonates with rhetoric about the needs for radical 
restructuring of engineering education and training, and the opportunities for enhancing 
learning through the use of new technologies. But while the volume of the standard 
rhetoric accurately reflects an urgently growing sense that learning in the new mille nn ium 
should be radically different, its content seldom reflects strategies and detailed, practical 
plans for effecting that change. Current research and development activities in learning 
technologies are characterized by being, a) fragmented and disparate, and b) mostly 
concentrated on near-term product development. 

Attempts have been made to remedy these problems - for example, NSF 
sponsored Engineering Education Coalitions, and the Learning Federation is developing a 
research agenda for learning technologies designed to have the greatest impact on post- 
secondary and life-long learning in science and technology. However, to date, they have 
not made a significant impact on engineering workforce training. 

To set the stage for succeeding presentations, an overview of the advanced 
learning technologies is given in this presentation along with a brief description of their 
impact on future aerospace workforce development. The presentation is divided into five 
parts (see Figure 1). In the first part, a brief historical account of the evolution of 
learning technologies is given. The second part describes the current learning activities. 
The third part describes some of the future aerospace systems, as examples of high-tech 
engineering systems, and lists their enabling technologies. The fourth part focuses on 
future aerospace research, learning and design environments. The fifth part lists the 
objectives of the workshop and some of the sources of information on learning 
technologies and learning networks. 
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EVOLUTION OF LEARNING TECHNOLOGIES 


Computer-based learning technology dates back to the 1960s (see Figure 2). 
Passive computer-based instruction systems were built in the 1960s and 1970s. Later 
developments in that period included learner modeling and more elaborate computer- 
learner interfaces. 

The addition of expert systems to computer-based instruction resulted in the 
Intelligent Tutoring Systems (ITS) of the 1970s and 1980s. These systems had explicit 
models of tutoring and domain knowledge, and were more flexible in their response than 
computer-based instruction. The advent of intelligent agents, which enabled the learner 
to manipulate cognitive artifacts from several perspectives or viewpoints, led to the 
Interactive Learning (IL) systems of the 1990s. In the late 1990s, a move towards 
Collaborative Distributed Learning (CDL), with distributed resources, occurred. Current 
trend is towards using Learning Networks (LN), for enhancing the effectiveness, access 
and affordability of learning. In learning networks extensive use is made of intelligent 
agents, learning is guided by cognitive systems, and the learner is an active and reflective 
participant in the learning process. 
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EDUCATION, TRAINING AND LEARNING 


There has long been a philosophical gap between education and training. The 
goal of education was to impart high-level cognitive skills that would underpin lifelong 
learning. The goal of training was to bring performance up to a level that would let 
people successfully achieve tasks. Recently, however, training began to emphasize the 
skills involved in lifelong learning, as evidenced by continual-growth workshops and 
online training facilities on the Internet. In a sense, both education and training 
objectives fit in the larger classification of learning objectives (Figure 3). 
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LEARNING OBJECTIVES, INSTRUCTIONAL MODELS 
AND TECHNOLOGIES 


The desired outcome of learning can range from information transfer to skill and 
knowledge acquisition to the more ambitious goal of development of critical thinking and 
creativity skills. The instructional model and method used for accomplishing these goals 
vary from instructor-centered, learner-centered to learning-team centered. In the learner- 
centered model, the learner is at the center of the learning process, and calls on many 
information sources. Learning-team center models include virtual classrooms and web- 
based distance learning models. The technologies employed in the three models are 
distribution, interactive and collaborative technologies, respectively (see Figure 4). 
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LEARNING NETWORKS 


The convergence of computing, communication and information technologies is 
providing opportunities for creating effective environments for life-long learning through 
expanding the concept of a university which is, typically limited to a campus, to that of a 
learning network (Figure 5). In such a network, the classrooms are augmented by e- 
leaming facilities (e.g., virtual classrooms); the libraries are expanded into intelligent 
knowledge repositories (with digital libraries and intelligent search and information 
visualization capabilities); the physical test and experimental facilities are augmented 
with access to more elaborate facilities at government labs, along with computer 
simulation of these facilities; and Immersive telepresence technology is used to provide 
interaction with geographically dispersed instructors and learners at other locations. 



Figure 5 
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ACTIVITIES OF LEARNING NETWORKS 


The learning networks can significantly enhance the effectiveness of engineering 
education, by changing the way three of the major functions of a university are carried 
out, namely, development of content for courses, packaging courses into curricula and 
programs, and delivery of these programs to learners (Figure 6). 

Each course is divided into self-contained learning modules, and a consortium is 
established for generating the best content for each of the learning modules. Advanced 
instructional technology; modeling, simulation and visualization facilities and authoring 
tools are used in the development of the modules. 

The learning modules are then packaged into disciplinary and interdisciplinary 
courses and training programs to satisfy the needs of diverse groups. 

The packaged modules are presented to individuals as well as groups of learners. 
Collaboration and interaction is made available at many levels, both synchronously and 
asynchronously. 
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ADVANCED LEARNING ENVIRONMENTS 


In order to meet the life-long learning demands of the future and broaden the 
awareness among the researchers and engineers in high-tech areas, three categories of 
learning environments are needed; namely, expert-led group learning environment; self- 
paced individual learning environment; and collaborative learning environment (Figure 
7). The three environments, in combination, can reduce the time and cost of learning, as 
well as sustain and increase worker competencies in high tech organizations. 

The human instructors in these environments will serve many roles, including 
inspiring, motivating, observing, evaluating, and steering the learners, both individually 
and in distributed teams. 



Figure 7 


9 






EXPERT-LED LEARNING ENVIRONMENT 


The human instructors in expert-led distributed learning in a virtual environment 
serve as coaches, guides, facilitators, and course managers. Their presentations focus on 
a broad overview of the topic and its diverse applications (Figure 8), and end with more 
penetrating, what-if questions that can enhance the critical thinking and creativity of the 
learners. Elaborate visualization and multimedia facilities are used in the presentations. 
Routine instructional and training tasks are relegated to the self-paced individual learning 
environment. 
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SELF-PACED LEARNING ENVIRONMENT 


The individual learning environment engages the learner and provides a high 
degree of tailored interactivity. It can be used for self-paced instruction of routine 
material not covered in the lecture. Using virtual instructors assigned by the human 
instructors can enhance such instruction. It can be used to study the physical phenomena 
occurring at different length scales using advanced visualization, multimedia and 
multisensory immersive facilities. The individual learning environment can serve to 
carry out numerical and virtual experiments - computer simulation of physical 
experiments (Figure 9). 
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SELF-PACED LEARNING ENVIRONMENT 
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COLLABORATIVE / DISTRIBUTED LEARNING ENVIRONMENT 


Collaborative learning environments teach teamwork and group problem solving. 
Instructors and learners can be geographically dispersed. Eventually, they can be brought 
together through i mm ersive telepresence facilities to share their experiences in highly 
heterogeneous enviro nm ents involving different computing platforms, software, and 
other facilities, and they will be able to work together to design complex engineering 
systems beyond what is traditionally done in academic settings. Because participants can 
be virtually co-located without leaving their industry and government laboratories, 
collaborative learning environments can enable the formation of learning networks 
linking universities, industry and government labs. The ultimate goal of these learning 
facilities is to create an intellectual environment where academic and experiential 
learning are effectively and efficiently co-mingled. In such an environment, academic 
rigor is learned in concert with professional job performance, and academic complexities 
are addressed within the industrial concern (Figure 10). 



Figure 10 
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COLLABORATIVE / DISTRIBUTED LEARNING ENVIRONMENT 



Figure 10 (continued) 
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SPECTRUM OF LEARNING ACTIVITIES 


The various learning activities can be grouped into three categories: learning 
science; learning technologies, infrastructure and tools and; learning systems (Figure 1 1). 
The first category include human cognition/cognitive learning and thinking processes; 
learning strategies, mechanisms and approaches; models of instructor/leamer interaction; 
knowledge representation and; evaluation, assessment and measurement issues in 
learning, and monitoring of learner behavior. The second category cover advanced 
simulation and interrogative visualization facilities; multimodal and adaptive interfaces; 
tools that support active engagement in tasks, collaborative and reflective processes; 
intelligent and mobile agents to provide help, motivation and move scenarios forward 
and; knowledge management, assessment and evaluation tools. The technologies and 
facilities can be integrated into the three learning environments: expert managed, self- 
paced and collaborative. The third category include e-leaming systems and learning 
networks. 
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RELEVANT TECHNOLOGIES 


The development of advanced learning systems and learning networks require the 
synergistic coupling of a number of enabling and leading-edge technologies. These 
include (Figure 12): novel computing technologies/paradigms; high-capacity 
communication/networking; human/computer/network interfaces; instructional; 
information/knowledge; intelligent software agents; modeling, simulation and 
visualization; cognitive systems and; human performance. 



Figure 12 
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GOVERNMENT AND NON-GOVERNMENT ACTIVITIES 


A number of government and non-government activities in the areas of advanced 
learning technologies and learning environments are currently underway. The 
government activities include (Figure 13): Advanced Distributed Learning (ADL) of 
DoD/OSTP; Adaptive Learning Systems of NIST; NSF Engineering Coalitions; e- 
Leaming of the Department of Education and; Distance Learning of FGDL. 

The non-government activities include MERLOT, FSRI’s Advanced Learning 
Environments; University of Mississippi’s Geospatial Information Coursework 
Development; MIT Open/Courseware Project; American Society for Training and 
Development (ASTD); United State Distance Learning Association (USDLA); Aviation 
Industry CBT Committee (AICC); and, The Training Place. 
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LEARNING TECHNOLOGY VENDORS 


A list of some of the learning technology vendors is given in Figure 14. These 
include WBT, Mentorware, Pathlore, Avilar, Elluminate, and the Training Place. 
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TECHNICAL STANDARDS 


Faced with a multiplicity of learning applications on the web, various initiatives 
can be found defining sets of metadata for describing them. It is of critical importance 
for the instructors and learners to find the learning materials when needed. This 
prompted a number of organizations to develop accredited technical standards, 
specifications, recommended practices and guides for learning technology. These include 
(Figure 15), the Sharable Content Object Reference Model (SCORM) of ADL; IEEE 
Learning Object Metadata (LOM) Standard and; the Learning Technology Standards of 
the Center for Educational Technology Interoperability Standards. 
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KEY COMPONENTS OF ADVANCED 
MODELING AND SIMULATION ENVIRONMENT 


The realization of the strategic value of modeling and simulation in learning 
systems requires an environment that links diverse teams of scientists, engineers and 
technologists. The essential components of the environment can be grouped into three 
categories: intelligent tools and facilities, nontraditional methods, and advanced 

interfaces (Figure 16). The three categories are described subsequently. 



Nontraditional 

iMethods 


Intelligent tools 
and facilities 


Advanceniintertacesi 


Figure 16 


21 



INTELLIGENT TOOLS AND FACILITIES 


These include high fidelity - rapid modeling, simulation and interrogative 
visualization tools, synthetic immersive environments; close coupling of simulations and 
experiments; computer simulation of physical experiments and remote control of these 
experiments. In all of these tools, extensive use should be made of intelligent software 
agents and information technology (Figure 17). 
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EXAMPLES OF FUTURE AEROSPACE SYSTEMS AND SOME OF THEIR 

CHARACTERISTICS 


The realization of NASA’s ambitious goals in aeronautics and space with the 
current national budget constraints will require new kinds of aerospace systems and 
missions that use novel technologies and manage risk in new ways. Future aerospace 
systems must be autonomous, evolvable, resilient, and highly distributed. Two examples 
are given in Figure 18. The first is a biologically inspired aircraft with self-healing wings 
that flex and react like living organisms. It is built of a multifunctional material with 
fully integrated sensing and actuation, and unprecedented levels of aerodynamic 
efficiencies and aircraft control. The second is an integrated human-robotic outpost, with 
biologically inspired robots. The robots could enhance the astronaut’s capabilities to do 
large-scale mapping, detailed exploration of regions of interest, and automated sampling 
of rocks and soil. They could enhance the safety of the astronauts by alerting them to 
mistakes before they are made, and letting them know when they are showing signs of 
fatigue, even if they are not aware of it. 
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ENABLING TECHNOLOGIES FOR FUTURE AEROSPACE SYSTEMS 


The characteristics of future aerospace systems identified in Figure 18 are highly 
coupled and require the synergistic coupling of the revolutionary and other leading-edge 
technologies listed in Figure 19. The four revolutionary technologies are: 
nanotechnology, biotechnology, information/knowledge technology, and cognitive 
systems technology. The other leading-edge technologies are high-productivity 
computing; high-capacity communication; multiscale modeling, simulation and 
visualization; virtual product development; intelligent software agents; reliability and risk 
management; human performance, and human-computer symbiosis. 
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THREE NASA INITIATIVES 


The realization of NASA’s ambitious goals will require a diverse, technically 
skilled workforce - a new generation of scientists and engineers who can work across 
traditional disciplines and perform in a rapidly changing environment. 

NASA has developed a number of new initiatives for assured workforce 
development. These include: University Research, Engineering, and Technology 
Institutes (URETIs), the National Institute of Aerospace (NIA), and the Hierarchical 
Research and Learning Network (HRLN) (see Figure 20). The overall goal of these 
activities is to strengthen NASA’s ties to the academic community through long-term 
sustained investment in areas of innovative and long-term technology critical to future 
aerospace systems and missions. At the same time, the three activities will enhance and 
broaden the capability of the nation’s universities to meet the needs of NASA’s science 
and technology programs. 



Figure 20 
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HIERARCHICAL RESEARCH AND LEARNING NETWORK 


The Hierarchical Research and Learning Network (HRLN) is a pathfinder project 
for the future aerospace workforce development. It aims at creating knowledge 
organizations in revolutionary technology areas which enable collective intelligence, 
innovation and creativity to bear on the increasing complexity of future aerospace 
systems. This is accomplished by building research and learning networks linking 
diverse interdisciplinary teams from NASA and other government agencies with 
universities, industry, technology providers, and professional societies (Figure 21) in 
each of the revolutionary technology areas and integrating them into the HRLN. 

HRLN is envisioned as a neural network of networks. It is being developed by 
eight university teams, led by Old Dominion University’s Center for Advanced 
Engineering Environments. 
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IMPLEMENT AION OF HIERARCHICAL RESEARCH 
AND LEARNING NETWORK 


The phases of implementing HRLN are shown in Figure 22. The first phase 
involves development of learning modules and interactive virtual classrooms in 
revolutionary technology areas, simulators of unique test facilities at NASA, and a 
telescience system - an online multi-site lab that allows real-time exchange of 
information and remote operation of instrumentation by geographically distributed teams. 
These facilities will be integrated into adaptive web learning portals in the second phase, 
which evolve into robust learning networks. In the final phase, the learning networks are 
integrated into the HRLN. 
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ADAPTIVE WEB LEARNING PORTAL 


The Adaptive Web Learning Portal being developed as part of the HRLN project 
has the following major components (Figure 23): 

■ Advanced multimodal interfaces, 

■ Knowledge repository, 

■ Blended learning environment incorporating the three environments: 
expert-managed, self-paced, and collaborative, 

■ Learning management system, and 

■ Customized collaboration infrastructure 




Figure 23 
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VIRTUAL CLASSROOM 


Online training and virtual classrooms are typically used to provide learning 
environments with custom self-instruction, flexible tutorial support, and choice of both 
the place and time of learning. Three categories of facilities are used in these 
environments; namely: instruction, including multimedia lectures, links to other 
resources and tools for searching, browsing, and using archived knowledge; 
communication, including email, UseNet, chat centers, video and Internet conferencing; 
and course management and performance evaluation (Figure 24). 
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INTELLIGENT DESIGN ENVIRONMENT 


The future design environment will enable collaborative distributed synthesis to 
be performed by geographically dispersed interdisciplinary/multidisciplinary teams. It 
will include flexible and dynamic roomware (active spaces/collaboration landscape) 
facilities consisting of (Figure 25): 

■ Portable and stationary information devices 

■ Novel multiuser smart displays 

■ Telepresence and other distributed collaboration facilities 

■ Novel forms of multimodal human/network interfaces 

■ Middleware infrastructures and intelligent software agents 
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OBJECTIVES AND FORMAT OF WORKSHOP 


The objectives of the workshop are to (Figure 26): a) provide broad overviews of 
the diverse activities related to advanced learning technologies and learning 
environments; and b) identify future directions for research that have high potential for 
future aerospace workforce development. The format included eighteen half-hour 
presentations in eight sessions. 
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INFORMATION ON ADVANCED LEARNING TECHNOLOGIES AND 
LEARNING ENVIRONMENTS 

A short list of recent books, monographs, and reports on advanced learning technologies 

and learning environments is given subsequently. 

[1] “ Distributed Education: Summary of a Six-Part Series ”, American Council on 
Education, Washington, DC, 2003. 

[2] Levine, A. and Sun, J.C., “ Barriers to Distance Education'' , American Council on 
Education, Washington, DC, 2003. 

[3] Katz, R.N., “ WEB Portals and Higher Education: Technologies to make IT 
personal", EDUCAUSE, National Association of College and University 
Business Officers, San Francisco, CA, Jossey-Bass, 2002. 

[4] Berg, G. A., “ Why Distance Learning - Higher Education Administration 
Practices", Greenwood Publishing Group, Inc., Westport, CT, 2002. 

[5] Johnstone, S.M., Ewell, P., and Paulson, K., “Student Learning as Academic 
Currency ”, American Council on Education, Washington, DC, 2002. 

[6] Katz, R.N., Ferrara, E.M., and Napier, I.S., “ Partnerships in Distributed 
Education" , American Council on Education, Washington, DC, 2002. 

[7] Eaton, J.S., “ Maintaining the Delicate Balance: Distance Learning, Higher 
Education Accreditation, and the Politics of Self-Regulation ”, American Council 
on Education, Washington, DC, 2002. 

[8] Hitt, J .C., and Hartman, J.L., “ Distributed Learning: New Challenges and 
Opportunities for Institutional Leadership" , American Council on Education, 
Washington, DC, 2002. 

[9] Laurillard, D., “ Rethinking university teaching: a conversational framework for 
the effective use of learning technologies" , RoutledgeFalmer, London, 2002. 

[10] Oblinger, D.G., Barone, C.A., and Hawkins, B.L., “ Distributed Education and its 
Challenges: An overview ”, American Council on Education, Washington, DC, 
2001. 

[11] Collis, B and Moonen, J., “ Flexible Learning in a Digital World: Experience and 
expectations" , Kogan Page, London, 2001. 

[12] Lockwood, F. (editor) and Gooley, A., “ learning ”, Kogan Page, London, 2001. 

[13] Gray, J.N., et. al., “Using Information Technology to Transform the Way We 
Learn", [http://dlist.sir.arizone.edu/archive/00000047/], 2001. 

[14] “ Distributed Education and Its Challenges: An Overview ”, American Council on 
Education, Washington, DC, 2001. 

[15] “Using Information Technology to Transform the Way We Learn", Report to the 
President, President’s Information Technology Advisory Committee, Panel on 
Transforming Learning, 2001. 

[16] “ Digital Libraries: Universal Access to Human Knowledge" , Report to the 
President, President’s Information Technology Advisory Committee, Panel on 
Transforming Learning, 2001. 

[17] Finkelstein, M.J., Frances, C., Jewett, F.I., Scholz, B.W., “Dollars, Distance, and 
Online Education - The New Economics of College Teaching and Learning", 
Greenwood Publishing Group, Inc., Westport, CT, 2000. 


33 


34 


Optimizing Individual Learning Performance with Multidimensional 
Evaluations and Adaptive Systems 


Nishikant Sonwalkar, PhD 
Massachusetts Institute of Technology 
Cambridge, MA 


Reprinted with the permission of the Massachusetts Institute of Technology. 


35 


36 


OPTIMIZING INDIVIDUAL LEARNING PERFORMANCE WITH 
MULTIDIMENSIONAL EVAULATIONS AND ADAPTIVE SYSTEMS 


Dr. Nishikant (Nish) Sonwalkar is principal educational architect at the Academic 
Media Production Services center. He has been providing pedagogical guidance and 
supervision to many projects developed at MIT. He is co-author of the MIT first fluid 
mechanics hypercourse. 
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AGENDA 


The technology enhanced learning is a multidimensional space that allows 
individual students learn with their own learning preferences. The individual preferences 
depend on the personality type that can be inferred using various Jungian psychological 
learning type descriptors. It is proposed by the presenters that correlations can be found 
between the personality type and the learning preference. The fundamental dimensions 
for the learning are, media, learning models, and interactivity. The main agenda of this 
presentation is describing a multi-dimensional evaluation model that can measure the 
pedagogical effectiveness of the on-line courses and measure individual performance. It 
is proposed that the correlations between personality types and learning preferences in the 
area of media, models and interactivity are then used to optimize the individual learning 
performance. 
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OVERVIEW: EVALUATION AND ASSESSMENT METHODS 


We begin with a brief survey of evaluation and assessment methodologies. The 
methodologies can be of represented in three categories, pretest, assessments and post 
test. The pre-test can help in defining learner profile in terms of personality types 
described by Jungian models such as Myers-Briggs, skills analysis and aptitude analysis, 
such as, SAT, GRE and finally skills gap analysis to ascertain the current level of skills 
for an individual. The assessment can be conducted during the learning activity for 
learning model preferences, attitude surveys, formative surveys, diagnostic tests on 
concepts, and course tests such as mid-terms and finals. The post-test, are performed to 
understand the summative effect of the learning and performance improvements. 
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PERSONALITY TYPE DISCRIPTORS: JUNGIAN MODELS 


Most widely adopted model for personality type test is Myers-Briggs. This test 
was developed based on the personality type descriptions proposed by Jung and is 
conducted in the form of a detailed survey instrument that leads to personality type 
descriptions in the form of Extraversion or Introversion, Sensing perception and Intuitive 
perception, Thinking judgment or Feeling judgment, and judgment or perception. Based 
on these parameters the Myers-Briggs test describes sixteen personality types. 



Jungian Models: MBTI 




. a m ■ ■ ■ ■ ■ * f • ■ 

■" Myers-Briggs Personality type descriptors 

♦ E Extra version or 
I Introversion 

Whether to direct perception judgment mainly on the outer world (E) or mainly 
on the inner world of ideas. 

♦ S Sensing perception or 
N Intuitive perception 

Which kind of perception is preferred when one needs or wishes to perceive 

♦ T Thinking judgment or 
F Feeling judgment 

Which kind of judgment to trust when one needs or wishes to make a decision 

♦ J Judgment or 
P Perception 

Whether to deal with the outer world in judging (J) attitude (using T or F) or in 
the perceptive (P) attitude (using S or N) 


Copyright® Dr. Nish Sonwalkar, MIT, 2003 


Figure 4 


40 


THE PERSONALITY TYPES BY MYERS-BRIGGS SURVEY 


This table illustrates the sixteen personality types that can be assigned based on 
the survey instrument. 
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THE LEARNING MODELS 


The learning models have evolved based on three psychological models, 
behavioral which is based on Skinners approach of stimulus/response approach, The 
cognitive psychology models such as induction, deduction and constructivist models 
proposing collaboration, incidental and self discovery models of learning. 


Learning Models 


a a ■ ■ f f » ■ ■ -f ■ ■ — 


Behavioral Models 
Cognitive Model 
Constructivist Model 



Copyright® Dr. Nish Sonwalkar, MIT, 2003 


Figure 6 


42 


LEARNING THEORIES INFLUENCES 


The learning theories that have influence the direction of technology enhanced 
and enabled learning platforms are: Skinners, behaviorist approach, Lev and Dewey, 
situated learning, Piaget and Brunner’s constructivist model, Spriro’s cognitive 
flexibility, Merrill et.al.’s component display and Gardner’s multiple intelligence theory. 
These references all point towards multiple ways individual prefer to learn. These 
models have also led to learning style inventories to define the individual preferences on 
how instructions and content are organize to match the individual preference to receive 
knowledge. 
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THE RESEARCH PAPERS ON LEARNING STYLES AND STRATEGIES 


Some of the seminal papers and references are given in this figure for those who 
would like to pursue study in the area of learning styles and models. 
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THE LEARNING CUBE: THE 3-D OF LEARNING 


The learning cube model is a attempt to propose an integrative multi-dimensional 
framework for design and development of pedagogically effective technology enabled 
and enhanced learning modules. The y-axis describes the media preferences, such as, 
text, graphics, audio, video, animations and simulations. The x-axis describes the 
learning model that are taken form all three psychological theories namely, apprentice, 
incidental, inductive, deductive and discovery. The z-axis describes various modes of 
interactivity, such as, intelligent feedback, remediation, discussion, dialogue and 
discourse. It is proposed by the presenter that all three dimensions play equally important 
role in enhancing the learning process. The pedagogical effectiveness then can be 
defined in terms of available choices to learners provided in these three dimensions. 
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THE FUNCTIONAL LEARNING MODELS FOR ON-LINE LEARNING 


The short definition of functional learning models that can be adopted for the on- 
line learning paradigm are provided in this figure. 
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THE AVAILABLE CHOICES IN THE 3-DIMENSIONS OF LEARNING 


Three primary dimensions for the pedagogy are defined in term of choices that 
can be offered to learners based on the media, models and interactivity. 
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MEASURING PERFORMANCE WITH KIRKPATRICK METHOD 


The performance measure are defined for the training program based on the four 
level analysis proposed by Kirkpatrick, that defines first level as the reaction of a learner 
towards the training program and their current performance, the level II instruments 
evaluate the learning outcome in terms of performance improvements and retention of the 
performance as a result of the training program the level II instruments evaluate the 
behavioral change subsequent to the training program for performance and retention of 
the behavioral change for performance and the level IV evaluates the organizational 
impact of the training program and evaluation of the return on investment (ROI) based on 
organizational performance. 
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PERFORMANCE OPTIMIZATION FOR INDIVIDUALS 


The goal for the individual performance optimization is offer on-line learning 
modules that lead to higher motivation by providing engaging learning interface. The 
adaptive learning model by delivering the learning modules that match the learning 
preferences then can minimize the time to learn. The learning that has engaging interface 
will also lead to longer retention for the learning. These three variables, motivation, time 
to master the concepts and retention once optimized for an individual will lead to desired 
improvement in the performance leading to measurable return on investment (ROI).. 
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ADAPTIVE LEARNING MODEL 


The optimization of the individual performance is achieved by providing 
continuous adaptive remediation of the conceptual difficulties. The remediation is done 
at the concept level. Each concept has a diagnostic test. The diagnostic test determines 
the level of mastery for a given concept. Based on the performance in the diagnostic test 
first the concept deficiencies are identified and then the remedial content is represented in 
different learning style. This is equivalent to rotating the learning cube until best learning 
style statistically identified for the learner. 
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PEDAGOGICAL EFFECTIVENESS OF ON-LINE COURSES 


The multi-dimensional evaluation can now be performed based on a simple 
probability tree diagram based on the assumption that media, models, and interactivity 
elements described in the learning cube contribute equally and learning events are 
mutually exclusive and equally likely. A simple probability distribution for the media, 
model and interactivity events then can be represented by summation of individual 
events. 


It is important to note that the pedagogical effectiveness depends on all three 
fundamental dimensions of the learning cube. 



Pedagogical Effectiveness of 
Online Course 

■ ■ m ■ f 1 f ■ * ■ •- 

Media: Text, Audio, Video Animation, Simulations 

Strategies: Apprentice, Incidental, Inductive, Deductive, Discoveiy 

Interaction: Feedback, Remediation, Discussion, Bulletin, Chat 


(Sonwalkar, Syllabus 2002) 
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5 Learning Styles 
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THE PEDAGOGICAL EFFECTIVENESS INDEX 


The Pedagogical Effectiveness Index (PEI) can be represented by a simple 
probability summation model based on the number of media elements, number of 
learning model elements and number of interactivity elements. The PEI index, therefore 
increases with the availability of the events. 


Pedagogical Effectiveness Index 



PEI = £S i*Pi + ZMj*pj + Xl k *Pk 

Where S - Style, M = Media, and I - Inter- 
action; the subscripts define the elements' 
ranges: i - 1 to 5, j - 1 to 6, and k — 1 to 5; 
ar d X represents summation. 


Pedagogical Effeciveness Index 
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PEI CASES 


Three cases to demonstrate how PEI model can be applied to evaluate 
effectiveness of the learning modules with the integration of different media, model and 
interactivity elements. 




II 

PEI Cases 

! 

1 

1 

i 

y 

Case 1: The PEI for a course with one media element, one learning 
style, and one interactive element will be: 

Igf 

PEI = 0.055+0.068+0.066 = 0. 189 


Case 2: The PEI for a corn's e with 4 media elements, 3 learning 

H§1 

styles, and 2 interactive elements will be: 

11 

PEI = 4*0.055 +3*068 + 2*066 = 0.556 


Case 3: The PEI for a course with 6 media elements, 5 learning 
styles, and 5 interactive elements will be: 

i 

PEI = 6*0.055 +5*068 +5*066 = 1.0 

11 
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EVALUATING LEARNERS EXPERIENCE: FIVE FACTORED SUMMATIVE 

INSTRUMENT 


The learners experience can be evaluated based on the five-factored summative 
evaluation survey instrument. The five factors include content factors, learning factors, 
delivery support factors, usability factors and technological factors. 


M75 

Influencing Factors: Learner 


experience 

a a a ■ ■ V f f ■ i ■ ■ 


k Content Factors 


” Learning Factors 

' \ ; 

” Delivery Support Factors 


s Usability Factors 

Ki 

« Technological Factor 


(Sonwalkar, Syllabus Jan 2002) 
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THE CONTENT FACTORS 


The elements of the content factors ascertain the content is prepared with high 
quality and is authenticated by the reliable sources. The content also includes various 
multi-media elements, presentation style and attribution to the source of information. 





Content Factors 


« Quality 

1 

” Authenticity 


« Validity 

j— i~ - 

« Media 

|| 

^if 

» Presentation 


« Attribution 
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THE LEARNING FACTORS 


The content has to be organized with concept map that defines the content with 
the correct concept structure, pedagogically driven instructional design developing 
content sequencing for different learning models described earlier, media enhancements 
to accommodate audio, visual and kinesthetic experience, interactivity with peers, 
professors and practitioners, intelligent feedback for remediation and revisions and 
collaboration with a team of co-leamers. 


\i~M 

Learning Factors 
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DELIVERY SUPPORT FACTORS 


For the on-line offering the delivery support systems are very important. The 
learning management system, content management, accessibility standards 504, 508 
needs to be implemented. Extensive reporting of the student activities is recorded by the 
delivery support systems. 


Delivery Support Factors 

— ■ 1 ■ i * i * i * — 

« User management (LMS) 

« Course content management 
« Accessibility 
« Reporting 
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THE USABILITY FACTORS 


The usability of the on-line module depends heavily on the graphical user 
interface, interactivity, clarity, chunk size and page layout. Learners input to these 
elements is important to determine success an on-line offering. 


ry n 

Usability Factors 
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« Graphical user interface 
« Interactive design 

pi 

« Clarity 
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« Chunk size 
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THE TECHNOLOGY INFRASTRUCTURE FACTORS 


The technology infrastructure in essential for the success of the course. These 
considerations include the network bandwidth, target systems, such as, mac, pc and unix 
operating systems, server capacity, browsers, IE, Netscape, AOL, and database backend, 
SQL, Oracle etc. The information technology department sets standards for these 
technology standards. 


... 

Technology Factors 
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" Network bandwidth 


! « Target system configuration 
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« Server capacity 
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« Browser sottware 

18 

L— j^4 

« Database connectivity 

03 

Copyright® Dr. Nish Sonwalkai', MIT, 2003 23 


Figure 23 


59 



THE FIVE FACTORED SUMMATIVE SURVEY INSTRUMENT FOR 
MEASURING USER EXPERIENCE 


A survey instruments is create with a Lickert scale of 0 to 4 for getting learners 
experience for the five factors described in the earlier figures. The survey instrument 
provides a summative score for the course in the range of 0-4. 


iim 





Figure 5: Summative evaluation instrument 


! 

for rating online courses 
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Reference: Syllabus, January, 2002 
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Figure 24 
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OVERALL PEDAGOGICAL RATING FOR ONLINE COURSES 


The PEI provides information about the how the course is designed and the 
summative instruments provides evaluation of the learners experience. The overall 
pedagogical experience can be defined as the multiplication of these two evaluations. 
The rating is in the 4.0 scale. 
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Overall Pedagogical Rating 
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Pedagogical Rating = PEI x Average Summative Score 
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OPTIMIZING INDIVIDUAL PERFORMANCE; DAT A-MININ G 


In order to understand and optimize the learners performance a data-mining 
approach is proposed. The data-mining approach includes click stream data, summative 
survey instruments, pretest-post test results, diagnostic performance evaluation data and 
user profile and personality data. Based on this data that is populated in a database with 
an entity relationship model, data mining approach allows discovery of patterns and 
associations. 


r~^| 

Data Mining Approach 
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ENTITY RELATIONSHIP MODEL: SCHEMATIC 


A simple entity relationship model is proposed that describes relationship as: 
student as attributes that are defined by the personality data, the students evaluates the 
experience by a the summative instruments and student also has a performance. 


Entity-Relationship Model 
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ENTITY RELATIONSHIP MODEL: RELATIONAL DATABASE SCHEMA 


The entity relationship model given in the previous figure can be translated to a 
database model with primary key and foreign key model for the relational model between 
student attributes and student response to on-line course with the student performance. 
The proposed database model is described schematically with the relational element in 
this graph. 



Entity-Relationship Model 
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EXAMPLE CASE-STUDY: VIRTUAL ROTAMETER EXPERIMENT 


The proposed data-mining model was applied at MIT on a course in the Chemical 
Engineering department. The course “Chemical Engineering Project Lab 10.26” was 
designed to provide a web-lab to calibrate rotameter (a devise that measures flow rate of 
fluids) based on the virtual water collected in a flask and a timer. The statistical analysis 
was conducted on the student who used the web-lab to understand the calibration process 
and those who used real laboratory (wet-lab) with actual instrument and water as a fluid 
used for calibrating rotameter. The results (published elsewhere, see reference 6), 
indicated for majority of the students the web-lab provided better learning then wet-lab. 
The experiment was conducted for two consecutive terms and data was analyzed using 
the data-mining approach to determine the association rules and decision tree based on 
the analysis of 



Example: 

A Comparison of Web-Based and Laboratory Learning 
Environments: Wet-lab vs Web-lab 
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ASSOCIATION RULES 


The data below indicates the association rules that were derived based on the 
data-mining analysis for the optimization of the learning experience using data-mining 
analysis. For example to rule 1 Indicates that the junior student had a good experience 
with the web-lab then wet-lab and there was 62% support for this association. 


Association Rules 






EjEQ 


1 

Lab Environment = Hyperactive 

Student Year = Junior 

13 

62% 

2 

Student Year = Junior 

Lab Environment = 
Hyperactive 

13 

50% 

3 

Experience with Calibration = "No' 
AND Student Performance = B' 

Student Year = Junior 
AND Lab 
Environment = 
Hyperactive 

8 

29% 

4 

Experience with Calibration = No’ 
AND Lab Environment = Actual 

Student Year = Junior 
AND Student 
Performance = B' 

6 

43% 

5 

Experience with Calibration = No’ 
AND Lab Environment = Hyperactive 

Student Year = Junior 
AND Student 
Performance = B’ 

8 

44% 

6 

Experience with Calibration = No’ 
AND Lab Environment = Hyperactive 

Student Year = 
Senior AND Student 
Performance = B’ 

5 

28% 
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THE DECISION TREE ANALYSIS 


The data-mining analysis was also used to derive the decision tree. The tree 
indicated how various parameters have an impact on the performance of the students. 
Decision Tree for Determination of Student Performance with Student Year, Preference 
for Lab Environment, Experience with Rotameter Calibration, and Supposed Ease of Use 
as Categories in shown in this figure. Regardless of the year of the student, the past 
experience with calibration of a Rotameter, the choice of web lab, and the notion that the 
use of the web lab would facilitate the use of the actual lab, the most likely score is a B, 
i.e., the student consistently performed better on web-lab environment then wet-lab 
environment. The only case in which this does not hold is for third-year undergraduates 
who were undecided about their preferred choice of lab. 



Figure 3 1 
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MAXIMIZING INDIVIDUAL LEARNING PERFORMANCE: THE ADAPTIVE 

LEARNING CYCLE 


Therefore, utilizing the adaptive learning technology and data-mining approach 
for optimizing the performance it is possible to control the quality of the learning 
experience to maximize individual performance. The learning cycle approach depicts 
utilization of learning with adaptive technology leading to an evaluation of the learning 
strategy that givens the best performance results and providing remediation and guidance 
for the revision at the end of each cycle to continuously improve the performance 
outcome. 
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Maximizing Individual Performance 



Learning Cycle Approach 




Pre-test 

i 






, • K $ 


1 

Post-test 
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PROPOSED HYPOTHESIS FOR LARGE SCALE EXPERIMENT 


The proposed hypothesis is to conduct a large scale statistical experiment to find 
correlations between the personality type, preferred learning model and the performance 
optimization. It is possible to populate large scale data on a diverse population and to use 
the data-mining approach to find the hidden associations that are strongly supported by 
the data and to use these association to provide a decision tree for extremely efficient and 
optimized learning/training paradigm for the optimum results in the shortest 
training/leaming time. 



Hypothesis: Personality>Leaming 
Model>Performance 
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♦Thinking-Feeling 

♦Judging-Perceiving 
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♦Learning 

outcome 

♦Knowledge 

retention 

♦Job performance 
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CONCLUSIONS 


The assessment of on-line education is a multi-dimensional problem 

The three dimensions of on-line learning are: Media, Models and Interactivity. 

The pedagogical effectiveness index (PEI) can measure the effectiveness of on-line 
course. 

Summative evaluation can indicate the effectiveness of a course based on learners 
experience. 

The overall pedagogical rating of the course depends on both the course effectiveness and 
learners rating of the course 

A systematic approach to assess online instruction is needed to determine if correlations 
exists intended learning variables 

An experiment is proposed to evaluate effect of the learner profile and learning model on 
learning effectiveness and performance improvements 
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Conclusions 



* The assessment of on-line education is a multi-dimensional problem 
« The three dimensions of on-line learning are: Media, Models and 


Interactivity. 

a The pedagogical effectiveness index (PEI) can measure the 
effectiveness of on-line course. 

a Smnmative evaluation can indicate the effectiveness of a course based 
on learners expeiience. 

a The overall pedagogical rating of the course depends on both the 
course effectiveness and learners rating of the course 
a A systematic approach to assess online instruction is needed to 
determine if correlations exists intended learning variables 
a An experiment is proposed to evaluate effect of tire learner profile and 
learning model on learning effectiveness and performance 
improvements 
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LESSONS LEARNED FROM THE PROGRAM IN COURSE REDESIGN 


Today using information technology has become an integral part of the life of 
colleges and universities across the United States to provide access for new students and 
in the teaching and learning mission. Most institutions, however, are finding the use of 
technology has become a costly addition as they have bolted it on to the traditional, class 
room based method of instruction. Thus information technology has added to the 
problem of rising costs in higher education. In addition, some research has shown that 
the use of information technology produces learning outcomes that are only as good as 
those in more traditional classes, rather than improving the quality of the learning 
experience; these findings are sometimes called the “no significant difference” 
phenomenon. 

Managed by the Center for Academic Transformation at Rensselaer Polytechnic 
Institute and funded by a grant from the Pew Charitable Trusts, the Program in Course 
Redesign is designed to address these problems of the growing cost of information 
technology and the need to improve the quality of learning simultaneously. This talk is 
an overview of this program including a brief review of the process, the institutions 
involved and the lessons learned. 


LESSONS LEARNED 
from the 

PROGRAM IN COURSE REDESIGN 

Carolyn G. Jarmon, Ph.D. 
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PROGRAM PURPOSE 


This project is designed to address the three major problems facing higher 
education today: Access, Cost and Quality. As noted previously, many already believe 
that it is possible to increase access to higher education using information technology. 
However, there are many who also believe that using information technology to increase 
quality of student learning necessarily also increases cost. 

Although much has been written about whether or not it is possible to both control 
costs as well as improve the quality of student learning, there have been no demonstration 
projects that actually put these hypotheses to test. The Program on Course Redesign was 
established to move promise to practice. The overarching goal is to encourage colleges 
and universities to redesign their approaches to instruction using technology to achieve 
cost savings as well as quality enhancements. Chosen in a national competition including 
hundreds of applications, the 30 institutions each received $200,000 awarded in three 
rounds of ten schools. 

Every project includes a rigorous evaluation plan focused on learning outcomes 
measured by student achievement. External national experts provided guidance to the 
project teams and review of the final plans to ensure that the results are both valid and 
reliable. Specific results are reported later. 


PROGRAM PURPOSE 

To encourage 
colleges and 
universities to 
redesign their 
approaches to 
instruction using 
technology to achieve 
cost savings as well 
as quality 
enhancements. 



$6 million 
over 3 years 


Figure 2 
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PROGRAM OBJECTIVES 


The program focuses on large enrollment, introductory courses because these 
courses reach large numbers of students at all institutions across the United States, use a 
more or less standardized curriculum and are a prime area of ineffective teaching. In 
many institutions, these courses are primarily taught by large numbers of adjuncts. These 
courses influence future majors and are feeders to a number of other disciplines. Finally 
large introductory courses absorb large amounts of resources. At the University of 
Wisconsin-Madison, for example, it costs over $1 million each year to teach Introductory 
Chemistry because the enrollment approaches 4100 students. In addition, at some 
institutions, these courses have large rates of drop-failure-withdrawal (DWF) and 
students have poor performance in downstream courses. Furthermore, many large 
enrollment courses are taught in multiple sections that experience significant course drift, 
meaning that some instructors cover all the material planned for the course, while others 
do not. Thus students move on to the next course with very inconsistent preparation. 

The Program in Course Redesign is organized to provide a systematic approach to 
redesign so that other institutions might model this process and to provide mechanisms to 
share the information and conclusions drawn across the 30 projects. Thus interested 
institutions have multiple successful models that show better methods for students to 
learn, while controlling costs using information technology. The Center for Academic 
Transformation also provides opportunities for project partners to share their findings 
with other members of the higher education community. 


PROGRAM OBJECTIVES 


Focus on large enrollment, introductory 
courses 


Teach institutions “howto” redesign 

Create a body of shareable information 
and practice 

Support communication and 
collaboration 1 i r 

Disseminate the results 


Figure 3 
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ROUND I INSTITUTIONS 


The next three figures give an overview of the kinds of institutions that are 
involved in each of the three competitive rounds. The first group, shown above, includes 
one community college and some state institutions. Most are research universities with a 
large quantity of resources and a generally higher state of readiness to engage in course 
redesign. 

All 30 institutions in the Program were trying to solve some problem that had an 
important impact on student performance at their campus. Among these problems 
addressed by the projects are high DFW rates, difficulty getting qualified adjuncts, long 
waiting lists for particular courses that are important for graduation, and inconsistencies 
of preparation of courses leading to multiple majors. As institutions were examining data 
collected about the top twenty-five, highest enrolled courses, they were able to identify 
the key problems affecting students in these large enrollment courses on their campus. 


ROUND I INSTITUTIONS 

IUPUI (Sociology) 

Penn State (Statistics) 

Rio Salado College (College Algebra) 

SUNY at Buffalo (Computer Literacy) 

U of Central Florida (American Government) 
U of Colorado-Boulder (Astronomy) 

U of Illinois-Urbana Champaign (Statistics) 

U of Southern Maine (Psychology) 

U of Wisconsin-Madison (Chemistry) 
Virginia Tech (Linear Algebra) 


Figure 4 
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ROUND II INSTITUTIONS 


These institutions are the ones selected in second round of competition in the 
Program on Course Redesign. Here, there are a large community college and two private 
institutions. Again there are also a large number of research universities who have a 
higher level of readiness and thus, are better prepared to do the kinds of large scale 
redesigns represented by this program. There is a good mixture of academic disciplines 
including Psychology and Spanish along with the quantitative and science areas. 


ROUND II INSTITUTIONS 

Cal Poly Pomona (Psychology) 
Carnegie Mellon University (Statistics) 
Fairfield University (Biology) 

Riverside Community College (Math) 
The University of Alabama (Math) 
University of Dayton (Psychology) 
University of Idaho (Math) 

The University of Iowa (Chemistry) 
University of Massachusetts (Biology) 
University of Tennessee (Spanish) 


Figure 5 
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ROUND III INSTITUTIONS 


The final group of ten schools includes the third community college, Tallahassee 
Community College, as well as private and state colleges. Again there is a predominance 
of research universities that have the higher level of readiness to participate in such a 
large scale initiative. This group of ten courses also includes a good representation of 
humanities subjects such as English Composition, World Literature, Spanish and the Fine 
Arts. Over the range of the thirty projects, the schools have tackled a variety of academic 
disciplines with a variety of models of redesign. 


ROUND III INSTITUTIONS 

Brigham Young University (English Composition) 
Drexel University (Computer Programming) 
Florida Gulf Coast University (Fine Arts) 

Iowa State University (Discrete Math) 

Northern Arizona University (College Algebra) 
Ohio State University (Statistics) 

Portland State University (Introductory Spanish) 
Tallahassee Community College (English Comp) 
University of New Mexico (Psychology) 

University of Southern Mississippi (World Lit) 


Figure 6 
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GRANT PROJECTS BY DISCIPLINE 


Among the 30 projects are a range of academic fields taught as introductory 
courses. These projects include math and other quantitative disciplines, humanities 
courses, social science and science courses. The benefit of this variety of academic fields 
is that they demonstrate that the approaches included in these redesign projects are 
effective for multiple academic areas, rather than just quantitative courses or science 
courses, as some would suggest. In addition, by tackling the various academic areas, the 
participating institutions were able to learn from each other and to compare what worked 
and what did not in the different fields. 
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GRANT PROJECTS BY 
DISCIPLINE 

MATH AND OTHER SOCIAL SCIENCE (6) 

QUANTITATIVE (13) American Government (1) 

Computer Psychology (4) 

Literacy/Programming (2) Sociology (1 ) 

Math (7) SCIENCE (5) 

Statistics (4) Astronomy (1) 

HUMANITIES Biology (2) 

English Compositions (2) chemistry (2) 

Spanish (2) ML 

Fine Arts (1) MK 

World Literature (1) ^ 'SSSSk 
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CLEAR PROCESS FOR REDESIGN 


Key to effective course redesign is a process that prepares institutions to move 
systematically and leads to the ability to compare the cost per student before in the 
traditional course and the cost per student in the redesigned course. Each institution 
addressed two sets of criteria designed to assess their readiness to participate in such a 
large effort. The Institutional Readiness Criteria focus on the institution’s strategic 
intent, personnel infrastructure as well as the ability of the institution to support a large 
scale redesign effort. The Course Readiness Criteria help an institution select a targeted 
course and examine the preparedness of faculty, students, and staff who will be involved. 
Subsequently the process involves a pedagogical and time analysis of what is currently 
done in the course and what options are available to improve student learning and reduce 
the labor intensity involved. The redesign team then carefully redesigns using the power 
of the information technology and crafts a doable plan. The full process and supporting 
tools are available on the Center’s website for use by anyone who is interested. 

Members of the Center for Academic Transformation provided extensive support 
in the planning, implementation and evaluation stages. This support was particularly 
important as none of the institutions had previously attempted such a redesign with such 
large numbers of students successfully. The support included both consultation and 
guidance regarding successful techniques and options when a participating institution hit 
a snag or problem. 


Clear Process for Redesign 

Step #1 - Evaluate Institutional Readiness 

Step #2 - Select the course or courses using 
Course Readiness criteria 

Step #3 - Determine the specific tasks 
associated with offering the course. 

Step #4 - Determine all personnel costs for 
this course expressed as an hourly rate. 

Step #5 - Determine each person’s time spent 
on each task. 

Step #6 - Calculation the total instructional 
costs for the course. 

Step #7 - Redesign the course by task and 
recalculate costs. 


Figure 8 
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LARGE NUMBERS OF STUDENTS 


The figure above shows the large numbers of students that are involved in the 
Program on Course Redesign. The students studying in the 30 courses totals over 50,000 
in one year. As noted previously, the goal of this program was to demonstrate that 
information technology allows scaling of benefits successfully - greater student learning 
and lower costs of preparation and delivery. These high numbers indicate that it is 
possible to redesign large enrollment courses and to improve the learning environment 
for students. 

Another criteria that was extremely important is sustainability. All of these 
redesigns will continue to have large numbers of students each year and the redesigns are 
structured so that they can be sustained with opportunities for continuous improvement 
and so that faculty can modify and improve the student experience. 


LARGE NUMBERS OF 
STUDENTS 

Round I 20,585 students annually 

Round II 14,119 students annually 

Round III 18,724 students annually 

ANNUAL TOTAL 53,428 Students 


Figure 9 
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SIGNIFICANT SAVINGS 


The major goal for this Program was to demonstrate that it is possible to redesign 
large enrollment introductory courses using information technology so that the quality of 
student learning increases (more on that later) and the cost of delivering that improved 
learning declines. All thirty institutions in the Program on Course Redesign reduced 
costs. The range is from 16% to 84%, totaling over $3.2 million dollars annually for 30 
courses. The large enrollments in the courses is a key factor in the ability to save money 
as the pedagogical improvements that also reduce costs are scalable. Consequently the 
reduction in cost-per-student is significant when many students are involved. 

Later, specific examples of methods used to reduce costs are included. These 
include reduction in faculty time by using the information technology to do the kinds of 
activities that are easily accomplished by the computer. In addition, some of the tasks 
needed by students do not require the expertise of faculty, but can be accomplished 
extremely well by other, less expensive personnel. 


SIGNIFICANT SAVINGS 



Round 1 

Round II 

Round III 

Range 

16% to 77% 

20% to 84% 

28% to 56% 

Average 

34% 

44% 

41% 

Total $ 

$1,006,506 

$1,043,821 

$1,195,028 


Annually 


Figure 10 
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LESSONS LEARNED 


Turning to specific lessons learned from this process, the Program on Course 
Redesign has identified common characteristics of redesigns that when used, provide a 
better learning experience for students and reduce the cost of delivering the instruction. 
In addition, these common characteristics improve the quality measured by greater 
student learning. Further the various models established via the 30 different projects 
provide a range of labor saving activities that can be successfully used in concert with the 
improvement in pedagogical design for greater learning. Finally the Program has 
established that the characteristics that improve quality also reduce cost. Thus 
institutions can design learning environments that simultaneously accomplish these dual 
goals. 


The next several figures provide specifics regarding each of these lessons. There 
are examples of various specific pedagogical and labor savings changes that institutions 
can make to reduce costs and improve the quality of the student’s learning experience. 


— 

— 


LESSONS LEARNED 

Common characteristics of redesigns 

Increase in Quality 

Variety of labor saving options 

Variety of cost models all produce savings 

Cost and quality are related 




Figure 1 1 
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REDESIGN CHARACTERISTICS 


The key idea found in all of the characteristics in this figure is active student 
engagement. Students need to be engaged both with the content of the course and with 
others participating in the learning experience. Those others include the instructor and 
the other students in the course. Active engagement means that students are no longer 
watching, but are participating. As one faculty member at Virginia Tech said, “Students 
learn math by doing math, not by watching math.” That observation is not limited to 
math; it applies to all learning and these ideas demonstrate it. 

In addition schools have included characteristics that increase the efficiency of 
learning for students. They can access learning materials 24/7 and receive the assistance 
they need when they need it, rather than during limited periods of time during the day. 
Students attend class for those learning activities that have been specifically considered 
as requiring face-to-face content; the redesigns move other activities online so that 
students who need to spend a lot of time on particular concepts can do so. Thus, the 
learning time is allocated by the student to those areas where they have difficulty learning 
and in those areas where the student already knows the information, they can move 
through more easily. 


REDESIGN CHARACTERISTICS 


Emphasize active learning 
rather than passive note-taking 

Promote greater student 
engagement with the material 
ana with one another 

Reduce number of 
lectures/class meetings 

Replace presentations with 
interactive software used 
independently and in teams 

Provide on-demand, 
individualized assistance 

Provide 24 x 7 access to online 
learning resources 

Improving the Quality of Student Learning 



Figure 12 
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REDESIGN CHARACTERISTICS 


This second group of characteristics emphasizes recognition that some students 
need more practice and feedback than others. Here the use of automated grading with 
feedback permits students to work as many problems or practice a particular learning 
experience as many times as needed. In addition, after students have identified their own 
learning styles, it is possible for them to know more about how much practice and 
feedback they need and the ways that they learn best. 

The use of differentiated personnel strategies permits students to receive greater 
assistance, at a lower cost. For example, at Virginia Tech in the Math Emporium, 
students receive individual help from faculty, graduate assistants and undergraduate 
assistants on demand when they have a problem. Also, they can join small groups with a 
tutor or work in pairs if that is desirable. Such strategies allow the response to come from 
a variety of personnel who assist at various levels and then are paid accordingly, rather 
than having a faculty member do it all. 

Finally, these redesigns are breaking the credit-for-contact model. The redesign 
teams have carefully considered what kinds of activities need face-to-face activity and 
which are better learning experiences for students when done online. Rather than start 
with the idea that face-to-face is better, these teams have carefully considered the 
learning outcomes desired and then designed learning options to increase student success. 


REDESIGN CHARACTERISTICS 


Emphasize practice, feedback, 
reinforcement 

Respond to differences in 
learning style 

Use course management 
software to monitor student 
performance 

Automate grading of 
homework, quizzes, exams 

Replace single mode 
instruction with differentiated 
personnel strategies 



Break the “credit-for-contact” model 


Figure 13 
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QUALITY OUTCOMES 


The primary measure of quality in the Program for Course Redesign is increased 
student learning. Learning gains were evaluated by measures such as greater 
understanding of identified concepts, greater increases in learning using pre- and post- 
tests and greater mastery compared with the traditional course. Schools also measured 
course retention, comparing the number of students who dropped, failed or withdrew 
from the traditional course with the DFW rate in the redesigned course. One excellent 
way to reduce cost is to reduce the number of students who repeat the course. 

Some institutions sampled student satisfaction via focus groups or student 
surveys. Where used, these provided information about how well the course was 
working and allowed for continuous improvement. Other institutions drew conclusions 
about the role of motivation and offered multiple approaches so that the student’s overall 
motivation level was higher. After analyzing a two-term course sequence, one institution 
found they had a disproportionate amount of information in one term, and much less 
content to master in the other. They were able to remedy this problem. 

Finally, the role of structure in redesigned courses is extremely important; these 
models are not self-paced. The design teams built in flexible structure with check-points 
or frequent assessments not only to allow students some flexibility to work more or less 
on a topic, depending upon their needs, but also to include frequent and useful evaluation, 
so that faculty and students know how well they are progressing. Then the faculty 
member can intervene if needed. 



Figure 14 
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DO STUDENTS LEARN? 


This figure provides several examples of improvement in quality of learning. 
Some demonstrate higher grades compared with students in traditional courses. Others 
show comparisons using pre- and post-test measures. Several also measured retention, 
the reduction in drop-failure-withdrawal rates. Given that all of these institutions have 
redesigned large enrollment, introductory courses, increasing retention is extremely 
important for continuing student success. 

At the University of Tennessee, there were a large number of students who could 
not enroll in Spanish class that prepares them for the second year; this was a major 
bottleneck course. Students are now studying Spanish by shifting all grammar and 
writing to the online environment. The face-to-face class time has been reduced from 
four hours per week to two hours. In addition, the in-class time is spent almost 
exclusively on speaking Spanish. Instructors no longer work on grammar exercises; 
these are all done using an online workbook that is automatically graded. Based on data 
collected from students who have taken the traditional model, the students in the redesign 
are demonstrating greater proficiency in speaking and faculty are able to focus on the 
more complex tasks that need their attention. Students are using the workbook more 
effectively and the overall result is a significant decrease in cost of the course from $109 
to $38 per student. In addition, with the new model, the University of Te n nessee is able 
to accommodate over 500 more students each year. 


DO STUDENTS LEARN? 

IUPUI redesign students had higher grades than 
traditional students and scored higher on a concept 
knowledge test. DFW rates dropped from 50% to 23%. 

Penn State redesign students outperformed the 
traditional group on overall posttest performance (66% 
vs. 60%). 

Rio increased retention from 59% to 68%. 

UCF redesign students increased content learning by 
2.92 points compared to traditional students’ 1.67 point 
increase. 

USM redesign students showed an increase in concept 
knowledge. There has been a 10 -20% reduction in 
grades less than C . 


Figure 15 
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DO STUDENTS LEARN? 


These are additional examples of the learning institutions found in their 
redesigned courses, compared with the traditional courses. At Fairfield University, the 
team have measured the learning in downstream or subsequent courses. They found that 
students retained content about genetics from the first year, as they entered the second 
year course. Thus, the faculty member no longer needs to spend significant time early in 
the course reviewing concepts covered in the first year; students are remembering the 
information. 

At Carnegie Mellon University (CMU), faculty found that after the redesign of 
introductory statistics, the level of statistical literacy has increased dramatically. While 
their students have always been good at calculating the statistic, faculty were concerned 
that they were not as capable at selecting the statistic. The CMU redesign uses 
SmartLab, an automated intelligent tutoring system that monitors student’s work as they 
go through lab exercises. Because SmartLab provides feedback to students when they go 
down an unproductive path and closely tracks and monitors a student’s progress, it 
provides the equivalent of an individual tutor. The redesign is preparing students to more 
effectively utilize their statistical learning. 

The University of Idaho modeled its redesign on the emporium model of Virginia 
Tech. Three math classes were redesigned and the average math class grades have 
improved. Students can work hard on topics they find difficult and spend less time on 
those that are easier for them. Students can get assistance when they need it; the outcome 
is extremely positive. 


DO STUDENTS LEARN? 


Fairfield U redesign students in Biology 
scored higher (88%) correct in a second year 
Genetics course compared with students in 
the old model (79%) and 4% more students 
selected biology as a major. 

Carnegie Mellon students can not only 
calculate the statistic, but also select it, 
demonstrating higher statistical literacy. 

U of Idaho students had higher average math 
grades in all 3 classes that were moved to the 
Polya Math Center. 


Figure 16 
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LABOR SAVINGS TECHNIQUES 


This figure shows multiple ways to reduce time spent developing and delivering 
the course. The goal is to shift the work from the faculty, who already know the material, 
to the students who need to learn it. The tasks that can be accomplished well by the 
technology are moved. The automated grading of homework, quizzes and exams 
reduces the time for faculty or TAs and provides faster, more timely feedback for 
students. At the University of Iowa, Introductory Chemistry included over 16,000 items 
to be graded in one term; four graduate teaching assistants (TAs) were assigned fulltime 
to this task and could not evaluate every item. Now all items are graded by the 
technology; four TAs are available to accomplish other tasks. 

The use of online materials for training both TAs and adjunct faculty has several 
benefits. The consistency of training is greater, as the resources are available every time 
someone new agrees to teach a course and when a TA or adjunct has a question. There is 
no concern about whether someone was told something important or whether they took 
good notes - all information is readily accessible. If there are changes, the resources can 
be easily modified and every one can be easily informed. 

Key to effective redesign is the course management system (CMS). Monitoring 
and tracking are major tasks with large student numbers. Some schools incorporated an 
automated email system to assist with feedback to students and provide faculty with lists 
of those who have not completed benchmark tasks or other scheduled assessments. 
Faculty or others can follow up and provide the intervention, either personal or online, 
that students need. This helps academic development and motivation. 


LABOR SAVING TECHNIQUES 

Reduce number of lectures/class meetings 
Replace presentations with interactive software 

Use course management software to monitor student 
performance 

Automate grading of homework, quizzes, exams 

Save class time by moving illustrations and assessments 
online 

Use online materials to train TAs or adjuncts 

Replace one -to-one interaction with peer interaction and 
other strategies 

Substitute cheaper, less expert labor 

Key Idea: Students Are Working - Not Faculty! 


Figure 17 
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REDESIGNMODELS 


These models emerged from the thirty redesign projects. They were not 
established in the grant; they resulted as the projects developed their design responses to 
the problems they were facing on their own campuses. All models are scalable and, to a 
greater or lesser extent, all leverage information technology so that it accomplishes tasks 
it is most suited to do such as monitoring student progress and accomplishment; provide 
practice, review and feedback as much as an individual student needs; and evaluation of 
quantitative or short answer assignments. The models also include faculty and other 
people doing what they do best - providing individualized assessment and feedback when 
appropriate. All the models involve active learning, so that the student move to the 
center of the learning experience from the more passive orientation. 

The model selected by a particular institution is related to the academic discipline 
of the course, the particular student audience and the preferences of the faculty involved. 



Figure 18 
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WHAT DOES THIS HAVE TO DO... COSTS? 


Coined by The Ohio State University, the term buffet model is appropriate 
because it effectively conveys the idea that students are drawing on an array of learning 
resources to achieve an established set of learning outcomes based on the student’s 
individual learning style. Although faculty have always known that all students are not 
alike in their learning styles, capabilities or previous knowledge level, it has been 
difficulty to provide the range of learning resources and experiences needed for a truly 
individualized learning experience. Information technology can radically increase the 
options for students. It is no longer necessary to treat all students alike; institutions can 
provide more individualized and personalized learning experiences. 



What does this have to do with 
improving quality and reducing costs? 


Figure 19 
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A BUFFET OF LEARNING OPPORTUNITIES 


The five characteristics are part of the buffet model. First students can use one of 
several online instruments to identify their learning styles and the results can guide 
faculty in designing the array of high-quality, interactive resources. Students can 
establish individualized study plans given their preferences for learning and the resources 
that are linked to the established learning outcomes for the course. Built-in, continuous 
assessment provides the feedback students need to make progress or return to a topic and 
study further and the technology can monitor and track the student’s activity. Finally the 
appropriate and varied kinds of human interaction can be planned and provided as 
needed. 


The student becomes the focal point in the learning experience. The quality of the 
experiences increases and the costs decreases for both the institution and the student. The 
institution can staff as needed by students at a level appropriate to the need and the 
student can spend his or her valuable time studying effectively those topics that they have 
not yet mastered. 


A BUFFET OF LEARNING 
OPPORTUNITIES 

Initial assessment of each student’s 
knowledge/skill level and preferred learning 
style 

Array of high-quality, interactive learning 
materials and activities 

Individualized study plans 

Built-in, continuous assessment to provide 
instantaneous feedback 

Appropriate, varied kinds of human 
interaction when needed 

Quality and Cost Factors Are the Same! 


Figure 20 
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OHIO STATE UNIVERSITY 


The next several figures focus on one example of the application of the buffet 
model. First it is important to understand the kinds of issues that Ohio State University 
(OSU) was facing in their introductory statistics class. This course is quite large and 
serves a variety of majors across the university. It had already undergone a redesign 
effort that led to increased costs and increased workload for faculty. There were many 
sections and the outcomes for students were extremely inconsistent. Students were not 
attentive and lectures were poorly attended. The DFW rates were too high - 20% of the 
students repeated the course each quarter. Further, faculty knew that some students 
successfully completed several modules, but did not finish the entire course on-time. 
Thus, each term they had to start over, but might again not reach a sufficient level of 
completion to earn a grade. In addition, the faculty knew that there were many different 
learning preferences and styles among the students enrolled. 


Ohio State University 
Statistics 

2850 students each year 

Previous redesign using IT increased the cost 

Too many emails for faculty 

Faculty time was used inefficiently 

Inconsistency among sections 

Lectures were poorly attended 

20% of the students repeat the course each 
quarter even though most have satisfactorily 
completed initial modules 

Students had highly variable learning styles 


Figure 21 
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OHIO STATE UNIVERSITY STATISTICS 


The Ohio State University (OSU) has developed a redesign project that uses these 
principles. OSU offers students a choice of interchangeable paths to learn each course 
objective. The buffet includes lectures, discovery laboratories, live and remote reviews, 
small group study sessions, videos, training modules, oral and written presentations, 
active large group problem solving, homework assignments (TA graded or self-graded), 
and individual and group projects. After an orientation to this new approach to learning, 
students assess their learning preferences and at the beginning of each module, enter into 
a contract in which they make choices among the options based on their understanding 
of their own style and preferences of learning. The contract provides students with a list 
of what needs to be accomplished with checkpoints and dates for completion. Based on 
their experience, students may make different choices in subsequent modules. Course 
software monitors their progress and provides management assistance for faculty. 


Ohio State University 
Statistics 

Students use online assessment by Felder 
and Solomon. 

There are multiple routes to established 
outcomes for each module. 

Students are assisted in thinking about how 
they approach learning and what mode is 
easiest for them. 

Students file a learning plan for each module. 

Various kinds of learning activities using 
websites, software, video lectures, small 
group discussions, individual and group 
projects. 


Figure 22 
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OHIO STATE UNIVERSITY STATISTICS 


There are some additional benefits for OSU in this buffet strategy. Several kinds 
of personnel participate, providing greater interaction with students and assistance when 
they need it. Graduate teaching assistants (TAs) as assigned tasks in the array of 
experiences based on their certified level of competence. The training materials for the 
identified levels are online and TAs can review and practice prior to demonstrating their 
ability. As TAs become more skilled, they move from grading and working with 
individual students or small groups to responsibilities for larger face-to-face groups or 
monitoring online components. Thus, TA development is an integral part of the buffet 
and provides continued quality control. 

OSU solved the dilemma of students who complete part of the five credits, but 
cannot finish the entire course in one term. Students who complete three or more 
segments earn credit for those segments and only re-enroll for the remaining segments in 
the next term. Thus, more students are ultimately successful, even though some may take 
longer than others to reach the level needed for success. 


Ohio State University 
Statistics 

Various kinds of personnel assist with the 
various learning activities including TAs, 
certified for various activities, and faculty. 

TAs are trained and certified to do various 
kinds of teaching such as grading, individual 
tutors, lab supervision, small group 
facilitation in person and online, and larger 
group facilitation. 

TA materials and training guides are online. 

If students don’t complete all five credits, 
they can re-enroll only for the part remaining. 


Figure 23 
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PROGRAM OUTCOMES 


The Program in Course Redesign has achieved its major goals. The 30 projects 
have produced multiple models for course redesign and these are useful to the academic 
community. The 30 projects have created a substantial body of practice and proof of 
concept that shows that these redesign models are scalable. All 30 projects saved money 
for their institutions and the amount saved is directly tied to the kinds of decisions that 
the redesign teams made. The Program has shown that it is possible to teach a retum-on- 
investment strategy and that institutions can implement it. The 30 projects demonstrate 
that it is possible to improve the quality of the large enrollment courses and that the 
multiple models can be used in range of academic areas successfully. Finally the models 
provide an alternative to the multiple-section model, and all of the difficulties related to 
consistency that these have. These represent the original goals and the Program has 
clearly demonstrated the successful proof of concept for others in higher education to use. 


PROGRAM OUTCOMES 

Producing multiple models for redesign 

Creating a substantial body of practice and 
experience that affects large numbers 

Teaching a “return-on-investment” strategy 

Improving a prime area of ineffective 
teaching, the large lecture course 

Creating an alternative to the “multiple 
section model” 


Figure 24 
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RESEARCH QUESTIONS 


These thirty projects have provided a proof of concept of two ideas that were 
previously considered polar opposites. Most believed that if an institution used 
information technology to improve the quality of a course, then the cost would, of course, 
increase. This Program has demonstrated that it is possible to decrease the cost of 
offering the course, while enhancing student learning. However, there is further work to 
be done. While these 30 institutions have been successful, many others are not ready to 
undertake such a large-scale course redesign - how can they increase their readiness? 

The buffet approach demonstrates it is possible to design learning enviro nm ents 
that can be individualized for students, but we have only two projects among the 30 that 
have done so. There are undoubtedly more options available and what are the next steps 
to expanding this concept and applying it in other academic areas? 

What kinds of models work best for what kinds of students and academic areas? 
Although there are some indicators of answers for these questions, greater study and 
research is needed. 

Finally, more software options are needed. The market for software is expanding, 
but the opportunities are much wider than the market currently allows. How can software 
be designed to meet the needs of different kinds of students and how can faculty know 
what to use with different students? These are among the valuable research questions - 
and there are likely many more! 


Research Questions 


How can we make sure institutions are ready 
for large scale redesigns? 

What are the next steps to expand the options 
for individualized learning environments? 

Which models work best for students with 
various learning styles and for particular 
academic areas? 

How can we improve the software options so 
that they meet the needs of students more 
effectively? 


Figure 25 
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LESSONS LEARNED FROM THE PROGRAM IN COURSE REDESIGN 


The Program in Course Redesign has a well organized and comprehensive 
website that is available to all who would like to explore these ideas further. Complete 
information about each of the thirty projects that make up this program is available on the 
Center’s website: www.center.rpi.edu. Every reader can easily explore specific projects 
that are particularly useful to an individual institution by visiting this searchable site. In 
addition, each of the projects includes a contact person, whose name and email address 
are easily found on the website. Thus, should a particular project be relevant to a 
problem under consideration, be sure to consult the contact person at the institution that 
has investigated the academic area of interest. These contacts are willing and excited to 
talk with anyone who would like to ask questions or learn more about a particular one. 
Do not hesitate to contact Dr. Carolyn Jarmon or the project representatives for more 
information. 


LESSONS LEARNED 
from the 
PROGRAM 

IN COURSE REDESIGN 

Carolyn Jarmon, Ph.D. 
Jarmoc@rpi.edu 

www.center.rpi.edu 


Center A , 

/..Academic 

Transformation 


Figure 26 
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LESSONS FROM A RESOURCE FOR LEARNING ACROSS DISCIPLINES - 
THE AEROSPACE DIGITAL LIBRARY 


Narayanan Komerath and Marilyn Smith 
School of Aerospace Engineering 
Georgia Institute of Technology 
Atlanta, GA 30332-0150 
Naravanan.komerath @ae. gatech.edu 

Website for users: Aerospace Digital Library http://www/adl.gatech.edu 

Research on engineering education: http://www.ae.gatech.edu/~nkomerat/ 

To work at the leading edge of technology, the engineer must be able to absorb 
and integrate knowledge from many disciplines, and from all over the world. The 
metaphor of a digital library was used to develop a self-sustaining resource prototype and 
study such learning. Conceptual design provides perspective through a learner-centered 
gateway, which accepts users at any level. Sequential course material presents the 
discipline rigorously and logically, with Concept Engines cross-linking subject areas. 
User experience and lessons learned are discussed. Simple introductions, undergraduate- 
level content and guidance to related areas are the most popular resources for users 
worldwide. Resources built around discipline portals offer natural vehicles for 
collaborative linking of knowledge bases around the world. 



Figure 1 
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OUTLINE 


Our aim is to develop an environment where aerospace engineers can integrate 
knowledge efficiently, bridging other disciplines as new knowledge is integrated into the 
knowledge base. We hope users from other disciplines will generate similar resources 
centered on their own disciplines. Here we list issues faced in developing such resources, 
then go on to describe our approach, experience with its growth, evolution and usage, the 
lessons learned, and guidance for the future. 


Outline 


• Dream: To learn - from the fundamentalsTa^he leading edge, as needed, where 
needed, regardless of “specialty” 

• Implementation: The Digital Library metaphor 

• Design-centered introduction as the portal to a discipline 

• Courses & Concept Engines 

• Technology in the Classroom 

• Experience 

• Example Application 

• Future 


Figure 2 
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LEARN - FROM THE FUNDAMENTALS TO THE LEADING EDGE 


In developing this resource, we have kept four different types of “customers”, to 
be satisfied in the short term, shown in the figure. We know of current participation by 
the first 3 types, and hope to bring in the fourth type in the short term. In addition, we 
hope that the resource evolves into a “one-stop engineering resource” which provides 
guidance to the latest developments in many disciplines, while maintaining the ability to 
learn the basics and the evolution history. In addition, the resource should cater to 
learners of different styles, and even select and organize material and resources to suit 
individual preferences and needs. 


Learn - from the fundarnentals to the leading edge. 

•Customer #1 : GTAE alumnus, 1 year out of school, faced with new assignment. 
“Class notes” too far away. Texbooks sold. 

•Customer #2 : Alumna, 10 years out of school. Needs quick brush-up of a different 
discipline to lead a cross-functional team. 

•Customer #3: Engineer anywhere in the world, any time, needs 
method in any other field. 

•Customer #4: Executive deciding on investment in a new technology. Seeks basic 
understanding. 

• Guided access to any technical field. 

•Learner-Adaptive interfaces. 

•Learner-Adaptive organization of material and resources. 






Figure 3 
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THE DIGITAL LIBRARY METAPHOR 


We used the “Digital Library”, drawing upon the ten essays in Ref. 1. The 
“library” metaphor includes archival collections, indexing, peer review, interactivity, 
gatherings, guidance and exchange between learners - a metaphor for the learning 
environment as well as content. As a starting point for content development in cross- 
disciplinary learning, users need basic knowledge to decide where to seek data. Thus we 
aspire to provide guidance and linkage, and leave content development to specialists. 

We have followed a bottom-up approach to system development. The initial 
critical mass of content was developed, based on available knowledge and hypotheses. As 
usage and users evolved, the lessons were used to refine the system and expand it. The 
picture shows aerospace engineering sophomores from Georgia Tech conducting a flight 
experiment on the NASA KC-135 Reduced Gravity flight laboratory in 1998 - a project 
which provided proof-of-concept for some of the cross-disciplinary learning hypotheses 
used in designing the knowledge resource. 


ThelDigtial Library Metaphor 

•Users need basic knowledge before seeking data 
Common resource for users from many disciplines 
The human mind is the best search engine of all! 

Informal, intuitive, private browsing and exploration 
•Access at the speed of thought 

•Meeting place for learners, teachers, disciplines and knowledge 

What type of content is best to start cross-disciplinary learning? 

• Basic knowledge, methods, data and usage experience. 

Provide guidance and linkage, and leave content development to specialists 



•Bottom-up approach to developing the system. Create - test - refine - expand 



Figure 4 
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IMPLEMENTATION USING THE “AEROSPACE DIGITAL LIBRARY” 


The learning resource was developed by adding basic engineering course content 
to the unique strengths of the internet. The content of present courses represent our best 
efforts, evolved through decades of experience, to provide a concise, logical, systematic 
and in-depth treatment of what we believe the learner must learn. It is true that these 
courses were developed under various constraints which may be removed by applying 
technology. However, until better systems are proven, these courses remain the baseline 
for knowledge resources. 

The structure of each course is linear and sequential, built up of distinct modules, 
or chapters, but each depending in some way on what was developed in prior modules. 
Using “Concept Modules”, we link concepts and methods across courses. We also tie 
professional applications and resources to these Concept Modules, and provide links to 
the leading edge of research. 

The Concept Modules with their links will keep evolving, and are the focal points 
for intellectual effort in developing this knowledge resource. As a starting point, the 
introductory lectures of each section in a course provides a rudimentary Concept Module, 
and provide the vehicles to link many types of resources. 


ImplementaWonJJsing the “Aerospace Digital Library” 

http://www. adl.gatech.edu 


Add basic course content to the strengths of the Internet: 

• Present different perspectives of the same material 

• ”Meta-courses" - supplement classroom time 

• Forum for technical discussions 

• Find & use knowledge across disciplines & levels efficiently 
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GATEWAY: DESIGN-CENTERED INTRODUCTION 


The Internet opens up avenues for cross-disciplinary problem solving. In 1996 
we used it to explain the diverse knowledge streams behind research projects. In 1997, an 
experiment in teaching first-term undergraduates demonstrated that the conceptual design 
of an airliner was an excellent focus to convey perspective on aerospace engineering 
(Ref.2). Conceptual design of a major system is an excellent integrator -when one 
realizes that it generally uses basic laws and thumb-rules to make decisions, leaving 
details to advanced methods and iteration. This makes it amenable and exhilarating to 
those who are yet to learn rationales for technological decisions. Seeing the areas of 
uncertainty helps learners understand the need for advanced theory and measurements 
that constitute much of the discipline. 
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APPROACH: THE RUNWAY ACROSS CANYONS 


The approach used in the Design-Centered Introduction is analogous to building a 
runway across land fissured by canyons or crevasses. Students are not assumed to have 
the time or ability to go deep into each area before climbing out again for perspective and 
going into the next area. Instead, we bridge the gaps in knowledge through thumb-rules 
or shortcuts - while providing the student a glimpse into the depths and possibilities of 
each sub-discipline. For example, spreadsheets are used instead of integration in 
calculating, say, the trajectory of a space launch vehicle - knowing that first-semester 
students may not yet have taken integral calculus. Drag coefficients are given for 
performance calculations. 

As the student’s perspective and confidence build up, along with the experience 
of locating what they need on the internet and elsewhere, the speed of learning does 
indeed increase dramatically, within a matter of days. E-mail interaction with the 
instructor, and construction of a web page are enforced at the beginning to get students 
used to the idea of using the internet and of asking questions. The content of the e-mails 
quickly evolves from “when is the deadline” and “I can’t get my web page to work” into 
technical decision-making questions indicating commendable depth of thought. What 
appears at first sight to be a compromise of academic rigor in fact enables a “takeoff’ into 
a level where such rigor is enhanced. 


Approach: The Ru nwa y Ac ross Canyons 
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LEARNER-CENTERED VIEW OF THE KNOWLEDGE BASE 


Our concept of the expanding knowledge horizons of a new learner in a discipline 
is inverted from the structure of specialization. Beyond the design-based portal is the 
core. We followed Rechtin’s synthesis approach (Ref.3)] to this problem with its infinite 
variety of paths to the content. This approach tackles the chicken-or-egg dilemma 
between creating content and structure for a large and unknown system. A prototype is 
created, enabling its performance to be studied in order to determine optimal structures 
for the full-scale system. 



Fluid Mechanics 


Physics 


Ethics 


Design- 
Centered 
Introduc- 
tion to 
V AE > 


Math 


Graphics 


Natural 

Science 


Flight Mechanics 


Learner-centered view of knowledge base 


Civil 


Chemical 

Engg. 




Mechanical 

Engg. 


Materials 

Engg. 


Electrical 

Engg. 


Industrial 

Engg. 


Computer 

Science 


#- £ 


, aws 


Figure 8 


110 






AEROSPACE DIGITAL LIBRARY PORTAL 


The front portal was designed to (a) provide fast access, and (b) to cater to 
different learner preferences and needs. Links were provided to the theory, archives, and 
other resources. A front panel provided links of interest to aerospace engineers, such as 
weather information, a Periodic Table, a Standard Atmosphere, and search engines for 
the Internet and the ADL. A selection of digital libraries from all over the world was 
linked. A Hyperbolic Tree site map serves learners who prefer a visual perspective of the 
ADL. Beyond this, two access methods were provided. The first was through the courses 
into higher levels of specific disciplines. The other was a table of subject areas. 
Resources were sorted into 9 different categories - as a starting point to hunt further with 
proper guidance. A Publication Archive was deployed in summer 2002. 
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OTHER DISCIPLINES 


Links to other disciplines of engineering and science are provided. As a start, 
these links are to the websites of schools dealing with these disciplines. We hope that 
dedicated gateways similar to our Aerospace gateway will be found in each of these 
disciplines in a few years, at which point the student can intuitively proceed from one 
discipline to another, gaining perspective as well as lucid introductions and expositions 
from the experts in each discipline. 
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TECHNOLOGY IN LEARNING 


The role of technology in the learning process is a subject of constant thought. 
Technology enables many dreams to be realized. One can revisit concepts and methods 
from many viewpoints, perform collaborative projects involving real-world complexity, 
learn in team environments, and cross boundaries between levels and disciplines 
seamlessly. These should of course enhance both vertical and horizontal integration of 
learning, and develop learners with much better perspective as well as grasp of the 
subject matter, enabling them to learn to solve problems anywhere. 

The role of technology inside the classroom is not so clear. We have long been 
skeptical of the usage of technology to demonstrate techniques which people used to 
learn by reading and experimentation before. Observations of Georgia Tech aerospace 
engineering students shows that many do in fact prefer simple, fast-loading websites, and 
are used to learning from textbooks and in-depth, detailed derivations. These aspects may 
be evolving as newer students arrive with a different set of aptitude, expectations and 
experience with using computers and the internet. 
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REALITY CHECK FOR COURSE REDESIGN 


Ideas such as “course redesign” and “technology use in the classroom” must be 
subjected to stringent reality checks. Alumni and employers expect that the value 
addition by a university education includes the experience of solving difficult problems 
under somewhat adverse conditions. While this is no excuse for persisting with 
needlessly adverse environments, it is important to ensure that as previous obstacles to 
learning are removed through redesign and technology, the students are not denied an 
once-in-a-lifetime opportunity to learn to deal with challenges. 

One criterion which has found acceptance is that of enabling the top performers in 
our classes to turn out better products and performance. The other criterion of reducing 
drop-out rate is more relevant to the first two years. These views often trigger negative 
reactions from education experts unfamiliar with the engineering leaning environment. 


tityiDheck for Course Redesign 

Must i be sure to understand goodpoints of current system before proceeding to change it. 

Employers value rigorous curriculum, tenacity, teamwork and flexibility. 

Alumni show an excellent record of success through evolving career paths. 

•Neither alumni nor employers really object to “Darwin ian^systems - “Is it as tough as the 
education I got?” 



Can the aerospace engineer who is taught using “advanced technology compeL 
one who learned through the “old, unkind” system? 

•“Average” performance better at mastering content? 

• “Top-performer” better? 

Should the measurement criteria be changed from “content mastery”? 

Does the “real world” REALLY accept such changes? 

- “Survival skills” of past generations must not be lost by new systems 

- As old problems are solved, new and more useful challenges must be introduced 
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EXPERIENCE WITH ADL 


The Aerospace Digital Library is in its fifth year of classroom usage at Georgia 
Tech. The resource is used by students at all levels - from first-year undergrads to PhD 
candidates. Types of usage are indicated in the figure. 


Experience with ADL 


Five years of Georgia Tech Aerospace Engineering classroom usage 

- as users, resources & expectations evolved^ 

• 1 st yr, undergrads: Average SATs ~ 1400, soTtrawith 1600 
High school GPA >3.5. 40% cite Space career motivation 

Sophomore through senior classes include roughly 50% with some “work” experience - 
Co-ops, internships, assistantships 
Graduate students from schools worldwide 
Approx. 400 users of ADL so far in formal courses. 

Courses & learners at all levels - Freshman through PhD. Theory, experiment, projects 
and competition teams. 

Usage in courses 

Lower-division classes use ‘net through cross-linked web-based notes and 
assignments requiring web searches - to take independent decisions 
Upper-division courses expect web usage to find published data and methods 
Graduate courses use web in finding and correlating 
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STUDENTS USE ADL IN MANY WAYS 


Project teams use the web site for “Knowledge Management”, storing documents 
for common access, and an electronic forum, maintaining continuity across years,. These 
students range far outside their home disciplines. Links to digital libraries on 
Shakespearean literature and classical poetry find usage, as do NASA resources. Seniors 
in aerodynamics classes cited discussions with faculty in Germany and Australia about 
computer codes. Codes from the University of Sydney, data from Japanese and Indian 
sites, and papers on flapping wing flight from England have all found usage. It is 
interesting to note that despite heavy usage in our own classrooms, 70% of total usage 
comes from outside Georgia Tech. 



Studerrt^use ADL in many ways 

•Project teams use the site for “Knowledge Management”. 

• Links to digital libraries and NASA resources. 

•Discussions with faculty across the world. 

•50% of students in courses using ADL go directly to ADL link; other 
50% will go to traditional paper sources first. T 

•Perception of deeper learning with web-based ADL material - 
reinforces lectures, provides different perspectives 

•Freshman course - students learn to cross disciplines easily 
part of conceptual design process. ; 

•“Lab” courses, design & independent projects provide cross- 
discipline experience. 

• Transfer from research to undergrad courses working well. 

• Reverse process may also be working - 

•Heavy usage as part of the courses; yet over 70% of total usage 
came from outside GT; 18% usage from outside the US. 
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LEARNING ON THE NET 


Asked how much of their learning was from each source, the near-unanimous 
answer was: 60% from attending class; 10-15% from the web-posted notes; perhaps 10- 
15% from out-of-class discussions and e-mail exchanges with the instructor, 10% from 
independent exploration, and 5% from the textbook. The web-based notes are still new 
enough that most students print them like class notes. The comfort level is not yet 
adequate to make the digital resource the primary means of learning - a necessary step 
for routine cross-disciplinary learning. Internet usage by project teams is heavy and 
ranges across areas where students have not had formal courses. Here, students gravitate 
to the “lighter” presentations, shunning derivations and in-depth explanations. Many 
learners feel overwhelmed by the Internet, and the challenge of prioritizing. Surprisingly 
few cited difficulties with nomenclature and terminology. The ability to study material 
through different approaches and nomenclatures is an encouraging sign. In no case has 
the availability of digital resources reduced in-class attendance. In fact, human interaction 
and guidance are appreciated more emphatically. 



^^Learilng on the Net 


• Where are they learning? 

- 60% of their learning was from attendtncLclass 

- 1 0-1 5% from the web-based notes 

- 10-15% from out-of-class discussions (emails, forums) 

- 10% from independent exploration (Internet and paper-b^sed) ► / 

- 5% from the textbook. 

-Why web-based notes? 

•“reference” about what they wrote down in class 
•clarifications on assignment 
•iterating between sources to study for tests 
•links to other web resources 

•learn better than is possible from the sequential “absolute” 
presentation in textbooks 
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CONCEPT MODULE PRESENTATION 


The idea of developing “Concept Modules” has found strong acceptance from 
learners. Opinion is divided on preferences for various approaches to developing these 
essays. As Concept Modules get more interactive features added, and enable input and 
evolution, they are more appropriately termed as “Concept Engines”. 


Concept Module Presentation 

Strong positive response at all levels for Concept Modules 

- Material presented in concise segments 

Classroom presentation and Concept Engines were presented in two styles for upper- 
level students: 

- Traditional classroom lectures with shorter web notes 

- Longer, more detailed web notes assigned as homework reading with lectures only to 
answer questions. 

• 1/3 of the students liked traditional lectures with shorter web notes, 1/3 liked detailed 
web notes as assigned reading, with lectures for question sessions. The rest were 
neutral. 

•Majority did not read ahead of the lectures, some said that “seemed to be a good idea 
for the future”. 

•For independent-thinking assignments - 50% utilized the web-based information first, 
other 50% went straight to the traditional library. 
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OBSERVATIONS ON WEB-ASSISTED LEARNING 


Present students still prefer to download web notes and use them like textbooks - 
they are not yet fully comfortable with learning straight off the internet. This inhibits us 
from extensive cross-linking - the downloading requirements would overwhelm students. 
Many already cite difficulties in prioritizing what to learn. 

Predictably, students tend to prefer the “lighter”, less mathematical treatments. 
However, it is interesting to note that treatments in different sub-disciplines, using 
different nomenclature, have not triggered any significant complaints from learners, when 
the context is properly explained. 

It is very interesting to note that even when the entire course notes are posted on 
the web, class attendance has not gone down - on the contrary, the value attached to 
human interaction in the learning process has gone up. 


Observations 


•Comfort level not yet adequate to make digital resource the primary means of learning 
necessary step for routine cross-disciplinary learning 

•Students gravitate to the “lighter” presentations 

•Many students print web site content like class notes 

•Some learners feel overwhelmed by the Internet, and by the challenge of prioritizing 
information 

Surprisingly few cite difficulties with nomenclature and terminology 
•In no case has the availability of digital resources reduced in-class attendance 
•In fact, human interaction and guidance are appreciated more emphatically 
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CROSSING DISCIPLINE BOUNDARIES 


People indicate a preference for a familiar interface. Thus learner-centered nodes 
are needed in each discipline, cross-linked to facilitate access. On the other hand, learner 
preferences vary greatly even between classmates - necessitating multiple structures for 
the links to the material. Given an essential core of in-depth material and broad concept 
explanations, it appears that users are happy to do their own searches, making it irrelevant 
whether the cross-disciplinary resources are linked or posted on ADL itself. This is a very 
encouraging finding on how people prefer to learn across disciplines. 


fssiocj Discipline Boundaries 


•Preference for a familiar interface : Learner-centered nodes are needed in each 
discipline, cross-linked to facilitate access. 



•Learner preferences vary greatly even between classmates - 
need multiple structures for the links to the material. 

•Given an essential core of in-depth material and broad concept explanations, 
users are happy to do their own searches. \ 

• Local storage unnecessary - cross-disciplinary resources are linked 
- encouraging finding on how people prefer to learn across disciplines. 
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LEARNING STYLES 


The existence of different learner styles is well-known in the literature. 
Differences in learner types have been cited even within a single discipline - thus it is 
certain that there will be wide differences between learner preferences when learning 
across disciplines. It may be impossible to design learning materials to suit all learner 
types simultaneously. An interesting possibility of web-based learning is that it is 
possible to tailor the presentation of given material, transparently, to the user - so that 
each user sees material organized according to the preferences that s(he) exhibits. 

Observation of students also suggests that engineering students will evolve in 
their learning styles, and in fact the same student may prefer to learn different subjects 
with different styles! 


earning Styles 
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INTERFACES TO SUIT LEARNER TYPES 


A set of possible metaphors for different learner styles in the field of aerospace 
engineering is given in the figure. 
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OTHER REACTIONS 


Reactions from the professional community show an interesting dichotomy. 
Alumni of engineering programs such as ours are positive and comfortable with ADL 
resources, expressing delight at desktop, on-demand access to course notes. These users 
have also contributed several items for informal publication, urging usage in classes. 
Usage by other professionals is not yet known to be significant. The idea of being able to 
find reliable in-depth resources and guidance on the web is new. Among other faculty, 
there is a dichotomy between those who appreciate, and others who reject, the necessity 
of teaching to solve problems across disciplines. Given that these are representative of 
engineering faculty, major obstacles remain in getting widespread faculty participation. 


.Other Reactions 


Engg. Alumni: positive and comfortable with ADL resources, cite desktop, on-demand 
access to course notes. 


Contribute items for informal publication, urging usage in classes. 

Other professionals: Usage not yet known to be significant. The idea of being able to 
find reliable in-depth resources and guidance on the web is new. 

Other faculty: Dichotomy between those who appreciate, and others who reject, the 
necessity of teaching to solve problems across disciplines. 
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STEADY-STATE SELF-SUSTAINMENT 


As we enter the 5 th year, we have shown steady-state self-sustainment at a 
moderate but useful level of existence and growth. Local storage is under 1.6 GB - a 
small fraction of the linked resources. In May-July 2002, the “background” usage - 
absent any courses requiring ADL usage - showed 135 unique users per day, compared 
to 87 in April 2001, with heavy undergraduate and graduate classes usage. Web-page 
requests per day have gone down from 3500 to roughly 1500. This is because our own 
students do more downloads in their courses. Also, a new site search engine has helped 
focus searches. The Design-centered interface is heavily used, as is the undergraduate- 
level course content. This is the most useful material to users across geographic and 
discipline boundaries. 


St eady-State Self-Sustainment 

5 th year of existence 

•Moderate but useful level of existence and growth. 

•Local storage is under 2 GB - a small fraction of the linked resources. 

•Design-centered interface is heavily used, as is the undergraduate course content. 

•Not practical to limit usage to “peer- reviewed” resources. 

Developing traits for independent validation of data appears to be most practical course 
at this time. 

Suggests need and role for different system of “peer reviewed Publication” where 
validity is established by open discussion and presentation of alternatives. 
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EXAMPLE APPLICATION: EDUCATION FOR A SPACE-BASED ECONOMY 


ADL is being used to help student teams engaged in developing a picture of the 
Space-based economy and its evolution - from the launcher industry of the 1950s and 
60s, to the exploration and research, and on to today’s communications industry, with the 
first space stations being developed. This is projected to extend, as refuelling and 
maintenance businesses develop, into development of lunar resources, habitats, a service 
economy, and eventually a full-fledged Space-based economy. NASA programs like the 
“NASA Means Business” program from the Texas Space Grant and JSC, and the NASA 
Institute of Advanced Concepts, provide the starting point for this effort. 
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EXAMPLE APPLICATION: “THERE’S SPACE IN YOUR FUTURE” 


Cross-Disciplinary knowledge resources permit better articulation of the 
opportunities related to Space-based businesses which can open up for people who 
consider themselves to be far removed from aerospace technology. 
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FUTURE DIRECTIONS 


Our efforts are shifting from resource acquisition to content refinement, cross- 
linking and adaptation to learners. It also appears that the “digital library” may be 
outmoded as a metaphor for an active learning environment, since the term “library” 
holds less meaning to many current students, who do their learning straight from high- 
speed internet links to their dormitory rooms. 

As alumni express the continued utility of ADL resources, it appears that the 
active engineering environment which provides impressive advantages to the modem 
engineer, is within reach. 


Future Directions 

•Shift focus from resource acquisition to refinement of content, cross-linking and error 
elimination 

Publication system for original work in a cross-disciplinary problem-solving context 

•Extensive indexing and cross-linking system using Concept Engines to enable users 
from distant fields to find each other through research motivatecN(om entirely different 
viewpoints 

• Meaningful Interactivity - Java applets and database access 

•Metaphor shift from “digital library” to “active engineering environment” suggested by 
usage patterns (“Library” is no longer a familiar term ) 

•The dream of the Modern Engineer, in an environment where all the data and calculation 
methods that one ever wants are accessible at the speed of thought, is beginning to look 

fpasihlfi 
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SUMMARY 


The objectives in the first phase are summarized in the figure. 


^^''\Summary 

Learner-centered resource for learning engineeringjundamentals and solving problems 
across levels and disciplines. Specific objectives were: 

• Develop core of fundamental knowledge on several disciplines accessible through the 
Internet, sufficient to form a useful learning resource for various levels. 

• Develop user experience in courses and research programs as a learning resource. 

• Assess learning methods and curricular structures enabled by these resources. 

• Test hypotheses about self-sustainment, growth and self-organization. 

First phase of the development - demonstrated self-sustainment and moderate growth in 
content and relevance to learners worldwide. 
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SUMMARY: WHAT IS ENABLED TODAY 


The ADL has already enabled 
as listed in the figure. 


several desired features 


of engineering education. 


~^Sumrnary: What is Enabled Today 

•First semester students do a credible job of “conceptual design”, integrating data and 
experience from all over the world. Replace&^overview” course 

•“Design” type large-system assignments with realistic parameters at all levels 

•Web-based reporting of assignments - iteration for improvement 

•Better vertical and horizontal integration - helps perspective at\al I levels 

•Communicates “expectations”, enhancing open-ended assgts (“The WOF^) 

•Courses improve with every teaching - “prior work” is online! 

\ ^ 

•“24x7” access to course notes, examples and references at every level 

•Size of resource now large enough to accommodate different learner styles 

•Usage by non-GTAE students now far exceeds captive audience usage - opportunity for 
use as a publication medium 

•Routine usage (e.g. of NASA TRs, TMs) in courses AND research through ADL 

•Absent-minded profs can forget their notes and not panic - can also tailor what students 
see & use for each assignment with “FAQs” 

• Growth of interesting sites and data found by students and alumni 

• NO dilution of the standards of thinking needed on tests and assgts 
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CONCLUSIONS AND ACKNOWLEDGEMENTS 


Our system approaches cross-disciplinary learning within the constraints of 
present curricular structure. We find it essential to take learners through familiar 
environments that resemble their courses and class notes. As internet-era students arrive 
in college, the comfort level with digital resources has risen. The challenge is to develop 
the willingness to delve into rigorous studies. Learners still must receive a great deal of 
support and guidance. Professional users from industry appreciate the convenience and 
depth of such resources, but stay within discipline constraints for the most part. It 
became apparent early that the unique value of the cross-disciplinary learning 
environment comes from 3 features: 

■ The core discipline material, with explanation and examples for 
understanding concepts and solving problems 

■ Selected links to outside resources 

■ The sorting of other resources into logical areas, and the guidance to the 
learner. 


'^^--^Conclusions 

Most sought by users across disciplines: basicUevel course notes 

Value in guided access to application data and results, and thence to sources of 
advanced knowledge 

Linked resources developed from course notes are seen to be valuable, when 
accompanied by clear explanations of concepts 

The idea of a discipline-specific portal providing access to the rest of the knowledge base 
has been demonstrated 

- and is ready for emulation in other disciplines to form a true cross-disciplinary 
worldwide engineering environment 
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THE INSTITUTE FOR ADVANCED EDUCATION IN GEOSPATIAL SCIENCES 


The Institute for Advanced Education in Geospatial Science (AEGS) is a project 
funded by NASA at the University of Mississippi. It seeks to support the NASA Earth 
Sciences Enterprise efforts to optimize NASA’s Earth Science investments for the benefit 
of the Nation. This presentation describes the basic work of the project, its goals, history 
and progress to date. The project is in year two funding of a potential five year funding. 



Institute for 
Advanced Education in 
Geospatial Sciences 


THE UNIVERSITY OF MISSISSIPPI 
Institute for Advanced Education in Geospatial Sciences 


Educating the Next 
Generation Of Scientists 

Dr. Pamela B. Lawhead, Director 



© 


Figure 1 


135 


THE PROJECT 


The project, as stated earlier is sponsored by NASA and is located at the 
University of Mississippi in Oxford, MS. It has two principal investigators with one of 
them, Pam Lawhead, serving as the Director of the Institute. 

The goal of the project is to create fifty online courses in Remote Sensing over the 
five year life of the project. Each year ten courses are put out for bid and the best ten 
submissions are accepted. This request for proposals insures that the course creators are 
content experts. “Equivalence of product” drives the online hosting of the courses. That 
is, we want the online presentation and delivery of each course to be as multi-media 
intensive as is effective. The goal is not to replace existing courses but, to provide 
courses created by content experts to as many colleges and universities as possible. This 
effort to create and host online courses has as its final goal the creation of a very large 
college educated workforce prepared to use the vast stores of information gathers by 
NASA and other remote sensing industries to enhance life on this planet. 


THE UNIVERSITY OF MISSISSIPPI 
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The Project 

Sponsored by NASA 

Located at the University of Mississippi 

Principal Investigators 

■ Pamela B. Lawhead - Computer Science 

■ Jay Johnson - Archaeology 
Courses created by content experts 
Multi-media intensive 

Goal: 50 courses in Earth System 
Science over five years 
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PROJECT GOALS 


Historically, little has been done to provide a consistent curriculum in Remote 
Sensing. This project has as its goal to do that first and then to move forward with its 
other efforts. To that end, we entered into a partnership with the American Society of 
Photogrammetry and Remote Sensing (ASPRS) to help us identify Academic and 
Industrial leaders. Once these individuals were identified, they were brought together to 
develop a robust, integrated curriculum. 

Once the curriculum was identified, courses were prioritized and sets of courses 
were designated for each year. By following the outline of the committee ten courses 
were created for each of the first two years of the program. The committee will meet 
again in October, 2003 to work on year three courses. Following this process either fifty 
courses or its equivalent will be written and enhanced for online delivery by the end of 
the funding period. 

In order to insure the widest range of acceptance and to provide a true equivalence 
of product a state-of-the-art course delivery system and creation process has been 
developed. This process is being tested as this presentation is being written. 
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Goals of the Project 

• To develop a robust, integrated curriculum to 
educate the next generation of geospatial 
information technology specialists. 

• To create a library of 50 university-level 
online courses reflecting a consistent 
curriculum in Remote Sensing 

• To develop a state-of-the-art course delivery 
system and course creation process that will 
be self-sustaining. 
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ALIGNMENT WITH NASA EARTH SCENCE SYSTEM’S NATIONAL 

APPLICATION AREAS 


NASA’s Earth Science Enterprise, Applications Program and Applications 
Directorate have identified three missions: 

“To develop a scientific understanding of the Earth system and its response to 
natural and human-induced changes to enable improved prediction of climate, weather 
and natural hazards for present and future generations.” 

“Expand and accelerate the realization of societal and economic benefits from 
Earth science, information and technology.” 

And finally: “To optimize benefits from NASA's Earth Science investments 

through systems engineering to advance decision support tools that serve the nation.” 

To be consistent with that enormous effort we have made very attempt to be sure 
that our courses are in alignment with these missions. The following figures 
demonstrate that alignment by listing the current and future courses that alignment with 
each of the identified application areas. 

The figure below shows the first of these areas and the courses consistent with 
Agricultural Competitiveness. 
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ALIGNMENT WITH NATIONAL APPLICATIONS 
AIR QUALITY MANAGEMENT 


In year one we have one course that meets this application area, Geospatial Data 
Synthesis and Modeling/ 

Year two will have two courses in this area. The authors of the courses Land Use 
and Land Cover applications and Community Growth will be selected at the June 9 th - 10 th 
meeting to be held in Oxford. 

Year three will have a course in Air Quality Management. The full description of 
that course will be written by the members of our National Advisory board in October. 
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WEATHER, CLIMATE AND NATURAL HAZARDS PREDICTIONS FOR 

AVIATION SAFETY 


The course that addresses this issue, Aviation Safety will be written in the 
October meeting and then put out for bids in the spring of next year. 
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PROPOSALS 


Once the course descriptions were completed a request from proposals was 
created and published. It was sent to the several professional organizations membership 
lists; it also appeared on our web site. Thirty proposals were accepted and the authors 
were invited to participate in a forum where the proposals were formally reviewed. Nine 
proposals were accepted and their authors were awarded sub-contracts of $80,000 each. 

The authors were then invited to a workshop in August where they were 
introduced to the course creation process and to their course liaisons. 


THE UNIVERSITY OF MISSISSIPPI 
I Institute for Advanced Education in Geospatial Sciences 

Request for Proposals 

• Advertised in Industry Journals 

• Sent to professional organization membership lists 

• Appeared on our Web Site - 

geoworkforce.olemiss.edu 

• 30 proposals submitted 

• 9 Awards of $80,000 made 
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THE COURSE FELLOWS 


The course fellows and their affiliations are listed below. The Lidar course was 
not awarded because there was not a strong proposal for it. 


THE UNIVERSITY OF MISSISSIPPI 
Institute for Advanced Education in Geospatial Sciences 

Course Fellow Awards 


Fellow: 

Arthur Lem bo, 

Cornell University 

Fellow: 

Rus Congalton, 

University of New Hampshire 

Fellow: 

Gouguing Zhou, 

Old Dominion University 

Fellows: 

Karen Seto and Erica Fleishman, 
Stanford University 

Fellow: 

Lori Bruce, 

Mississippi State University 



Fellow: 

James Campbell, 

Virginia Tech 

Fellow: 

Richard Forster, 

University of Utah 

Fellows: 

Conrad Bielski, JPL 
and Khaled Hasan and Greg Easson 

Fellow: 

Lynn Usery, University of Georgia 

Fellow: 

John Jensen, 

University of South Carolina 


, UM 
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THE COURSE CREATION PROCESS 


In order to build a course on line the Institute provides graduate students in a 
technology lab for each course fellow. The fellow is responsible for the course content 
only. That content may be delivered to the center in any way that the fellow feels 
comfortable. We have set up an online delivery process but some fellows found it 
difficult to use or preferred to give us the material in other ways. Once the material is 
presented to the student liaisons, it is enhanced, broken up into acceptable sizes and 
hosted online. The student liaisons work very closely with the fellows. We also provide 
an educational consultant for the fellows. This person is responsible for guiding the 
fellows in the development of very detailed course concept hierarchy maps. 
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THE PROGRAM DELIVERY SYSTEM 


On the programming side the course delivery system is divided into four parts, the 
user interface, XML/XSLT templates, publishing framework and the backend database. 
These are used to produce each course page. 


THE UNIVERSITY OF MISSISSIPPI 
Institute for Advanced Education in Geospatial Sciences 



Diagrammatic View of Current Implementation 



Java Server Pages 
(JSP) contains both, 
JSP code and XML 
(data). 


Tomcat evaluates the JSP 
and the Cocoon publishing 
framework is used to 
transform the XML into 
HTML using XSLT. 
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THE COURSE DELIVERY SYSTEM 


The delivery options for each course are varied. A student may enroll at any 
college our University, anywhere and access the courses. The home institution licenses 
the courses from the Institute. The home institution awards the credit to the student, 
gives grades and basically oversees the administration of the course locally. 


THE UNIVERSITY OF MISSISSIPPI 
Institute for Advanced Education in Geospatial Sciences 

Delivery Process 

Options Include: 

• Students enroll at home 
institution 

• Individual enrollment 

• Tuition paid to credit granting agency 

• Credit granting agency pays fee to UM 
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CURRENT STATUS 


We have currently hired a staff of four, a director, an associate director, a project 
coordinator and a network administrator. We have thirty graduate students at work. 
These students are divided into five teams and within those teams have multiple 
responsibilities. Some students are members of multiple teams. Student liaisons are 
responsible for identifying work to be done for each course and then parsing the work out 
to others. All students are members of the research team and expected to be actively 
involved in some research topic related to the course creation/ delivery process. 
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CURRENT STATUS 


As discussed earlier, the Advisory Board is an active part of the Institute’s work. 
The technology lab is established with students working actively each day. Two 
prototype courses are complete. Two other courses are 90 % complete and all others are 
approximately 60% complete. One short course is in process on Decision Support 
Systems. 

We work very closely with our educational expert in directing the overall design 
of each course. We have a very large group of graduate students at work and are, for the 
summer, using some undergraduates. 


THE UNIVERSITY OF MISSISSIPPI 
Institute for Advanced Education in Geospatial Sciences 



Current Status 

• National Advisory Board actively guiding the process 

• Course creation lab established 

• 2 Prototype courses complete minus quizzes 

• 11 Course Fellows under contract 

• 1 Short Courses in process 

• Consultant on Pedagogy directing course design 

• 31 students at work on animations and course 
delivery process 
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STATUS 


The Knowledge Engine, described earlier is complete and undergoing testing at 
this time. The Virtual Portal is being completed, including the Virtual Campus and the 
browser application. 

The course fellows are all under contract, the second round of RFPs will 
culminate in a conference on June 9 th of this year. The goal is to have six courses out for 
testing by the end of this summer. 


THE UNIVERSITY OF MISSISSIPPI 
Institute for Advanced Education in Geospatial Sciences 



Current Status 


Knowledge Engine Design Complete 
Virtual Campus delivery expected 
May 2003 

Browser with Renderware® successfully built 

Course Liaisons in place 

1 st Set of Course Contracts in place 

Concept Maps, first milestone delivered 

Mid ^ear course evaluations complete 

2 courses complete 

RFP for 2 nd round of courses finished 

Memorandum of Agreement signed with Boeing 
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RESEARCH RESULTS 


The articles listed below are evidence of the effectiveness of the work of the 
research team. This is the status as of April, 2003. There are many more papers under 
construction at this time. It should be noted here that, because of our successful 
International work we have had a name change. In the research papers our name appears 
and the Center for Geospatial Workforce Development (GWD) we are now the Institute 
for Advanced Education in Geospatial Sciences. 


THE UNIVERSITY OF MISSISSIPPI 
Institute for Advanced Education in Geospatial Sciences 


Current Publications By The Institute 

Liang Feng, Application of Database Technologies in the GWD 
Online Course Creation Project, Conference Name: The 2003 
International Conference on Information and Knowledge Engineering 
(IKE'03) Feb. 17, 2003 

Hong Zhou, Naive Bayesian Classifier for Microarray Data. 

International Joint Conference on Neural Networks, July, 2003 

Jun Liu Three-Dimensional Effects Created by Macromedia Flash 

MX, The 2003 International Multi-conference in Computer Science and 
Computer Engineering Draft Due: Feb 17,2003. 

Jie Tang Follow the Line: using vision in an autonomous mobile 
robot. The 4th British Conference "Towards Intelligent Autonomous 
Robots" (T1MR '03) Draft Due: 4/25/03 
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MORE RESEARCH PAPERS 


These paper topics indicate the scope of the work being done. The paper by Bhatt 
explores the work of using Virtual Reality to provide online laboratories in GeoSpatial 
Sciences. The others all have to do with technical issues regarding the delivery of the 
online courses or the creation process itself. 
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@ 


Parishweta Bhatt Virtual Laboratories Using Interactive 3-D 
Simulations for Remote Sensing Education ASPRS: May 2003 

Pushpa Grandhi Software Verification Tool for Ensuring 
Accurate Animation Delivery ASE(The IEEE International Conference 
on Automated Software Engineering) Submitted May 6, 2003 

Krishna Sajja Providing Streaming Capabilities to Online 
Applications CRIWG 2003, 9th Int'l Workshop on Groupware, 28 Sept.- 
2 Oct. 2003 

Manirupa Das Information Storage and Management in Large 
Web-based Applications using XML CCSC - MidSouth Conference, 
Memphis, Presentation: March 28, 2003 


Figure 17 
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Jianrong Yu Data Mining with Neural Networks Under a High 
Volume Noisy Environment Target Journal: Journal of Neural 
Computing and Applications, Submission Date: June, 2003 

Brenteria Travis Evaluation of Load Testing Software Packages 

Target Conference: International Conference on Software Testing and 
Analysis. Abstract Deadline: May 26, 2003 

Eric Windham Shockwave vs. VRML, ACM Symposium on Virtual Reality 
Software and Technology 2003 Draft Due: May 31, 2003 

Raju Vuppala Random Generation of Questions based on 

Navigational History Target Conference: Conference on Information 
Technology, Submission Date: June 20, 2003 

Ravi Darbhamulla Proxy Expert - Using the Student Support 
Monitoring Tool Conference: ITICSE 2003 


Figure 18 


150 


PUBLICATIONS CONTINUED 


This list continues the papers written by members of the research team. The list 
goes from policy issues to Adaptive Learning. Again, it is important to note that these 
papers are a reflection of the issues that we work on at the Institute and are critical to the 
success of the projects. 
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Manirupa Das Emulating the Real - World Expert for Online Course 
Delivery using Student Status Monitoring, Innovation and Technology 
in Computer Science Education mCSE - 2003, 

Keyur Patel Technical and Policy issues for Online Reserve Material 

Anchuri Pratap Secure Data Presentation 6th Information Security 
Conference (ISC'03) Submission Deadline May 1, 2003 

Jim Zhu Pricing for Web Services The 2003 International Workshop on 
Mobile Systems, E-commerce and Agent Technology (MSEAT'2003) 
Submission Deadline May 15, 2003 

Shrinivas Chappidi Random Algorithms for Quiz Generation (In 
Progress) 

Marcus Golden Adaptive Learning in an Online Environment (in 
progress) 
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RESEARCH CONTINUED 


Mr. Paris is the team lead on the Virtual Portal, he also is doing significant work 
on data compression both for course image delivery but for compression used in other 
areas as well. 
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Luis Paris 

1) An empirical study of variable-length codes for efficient storage of 
data. To be published in the "ACM Computing Surveys" journal 

2) Data compression methods for online delivery applications 

Submitted: "ACM Computing Surveys" journal 

p) Hosting a Virtual World in an online environment" To be presented in 
the CRIWG (International Workshop on Groupware). France, Sept 28 - Oct 
02, 2003 

ft) Introducing interleaved start-step codes for fast variable-length 
coding. To be presented in the DCC (Data Compression Conference) 2004 
Utah, March, 25 - 27, 2004 
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COURSE LIST 


The list below is the set of courses that are currently under construction. We 
expect to have courses, 3, 4, 5, 7 and 10 ready for delivery by the beginning of the Fall 
semester. 
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Courses for Year One 


1. Introduction to Geospatial Information 
Technology 

2. Sensors and Platforms 

3. Photogrammetry 

4. Remote Sensing of the Environment 

5. Digital Image Processing 

6. Advanced Digital Image Processing 

7. Aerial Photographic Interpretation 

8. Information Extraction using Microwave Data 

9. Information Extraction using Multispectral, Hyperspectral and 
Ultraspectra I Data 

10. Orbital Mechanics 

11. Geospatial Data Synthesis and Modeling 
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YEAR TWO COURSES 


The descriptions and the RFP for the courses listed here can be found at the 
Institute’s web site: http://geoworkforce.olemiss.edu . 

The meeting to determine the authors for each of these courses will be held on 
June 9, 2003. The fellows will then return to the campus for a workshop in August to 
begin the course creation process. 
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Courses for Year Two 


Advanced Sensor Systems and Data Collection 
Advanced Photogrammetry 

Information Extraction using Thermal Infrared Data 
Land Use and Land Cover Applications 
Smart Growth and Urban Regional Planning 
Applications 

Ecosystems Modeling Applications 
(GAP, biodiversity,fish/wildlife) 

Water Resources Applications 
Forestry Applications 
Mapping (Topographic) 

Business Geographies (industrial site location, 
banking, real estate, simulation and video 
games and individual) 





Figure 22 


154 


INTERNATIONAL EFFORTS 


Round one of the course selection process had applications from Greece and 
Canada. This was only the beginning of our International effort. We spoke to the ISPRS 
Council and Commission heads in December where we received their endorsement. We 
are in the earliest stages of working with CEOS in an effort to achieve a global market. 

Year three will be very involved in “internationalizing” the curriculum to insure 
that it has as large an audience as possible. 
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International Efforts 


Round one applicants from Greece and 
Canada 

Invited to present to ISPRS Council and 
Commission Heads 
Received endorsement from ISPRS in 
December 

AEGS became ISPRS's response to CEOS 
educational initiative 

International Advisory Board to meet in Fall 
to assure "internationalization" of Model 
Curriculum 

Second Round RFP to be delivered 
internationally through ISPRS Commission 
Heads 
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OUTSTANDING CHALLENGES 


The Institute still faces many challenges, none insurmountable but all are there 
before us. We are in the process of completing the licensing process. This requires a 
formal document which must be acceptable to our own institution as well as the 
subscriber institutions. It is being reviewed as this is being written. 

Web deliverable 3-D has a significant overhead that must be dealt with if the 
speed is to be acceptable. We are working on this on many fronts. We are working to 
reduce the files size, to compress and decompress the files, to use streaming and caching 
techniques. 

We are testing the delivery process on many levels, not just the speed but also the 
understandability of the interface as well. 

It is our hope that the second version will be filled with existing learning style 
technology. 
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Institute for Advanced Education in Geospatial Sciences 

Outstanding Challenges 

Technical % 

• Licensing Process^ 

• Web Deliverable 3-D Overhead 

• Testing of Delivery Process 

• Addition of Learning Style Technology 


Non Technical 

• Increasing Participation in RFP Response 

• Internationalization of Course Offerings 

• Establish Formal Relationship with CEOS 
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THE VIRTUAL PORTAL 


The virtual portal is nearing completion. We have the Portal sending information 
to the Browser but we must complete the loop and have the information flow in both 
directions. This is simply a programming issue and it is being solved. 



Figure 25 
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THE FINAL PRODUCT 


The image below reflects the final project. In it there is a standard browser, a 
video chat space, a white board, the 3-D virtual portal, a note pad and a list of who is 
online as the student is logged on. This will be the space used by the students. 
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THE WEB PAGE 


This is a snapshot of our web page which can be viewed at 
http://geoworkforce.olemiss.edu 
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LATEST NEWS 

June 9-11, 2003 - Second Annual Course Creation Fellows Symposium. Agenda 

Here! 


IAEGS EVENT OF THE YEAR! 

The Institute for Advanced Education 

Mississippi, is proud to organize and present 


e Second Am 


at toe University of 

;e Creation Fellows 



This Symposium will showcase selected candidates from institutions worldwide, who will 
compete for Ten New Course Creation Fellowships in Geospatial Science. Selected Course 
Creation Fellows will participate in generating a repository of online courseware to cultivate a 
highly skilled workforce to lead toe development of Geospatial Information Technology into toe 
next century. 

"We do not advocate replacing the traditional University setting, but enhancing it 
by advancing knowledge without boundaries, via Distance Learning." 

- Dr. Pamela Lawhead, Director 


Contact us at: Phone - (662) 915-3900; Fax - (662) 915-3901; Email - 
maten@olemiss .edu 

Best viewed at oi above a resolution of 1024 x 768 in IE ver5.5 upw. 


Phase II Course Opportunites 

1 Adv. Sensor Systems & Data Collection 
2. Advanced Photoarammetrv 


5. Community Growth 

6 Ecosystems Modeling Applications 

7 Water Resources Monitoring and Mgmt 
S Forestry Monitoring and Management 
9 Topographic Mapping 

10. Business Geographies 
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Reading, Learning, Teaching 


Lisa Ferro, Warren Greiff, Lynette Hirschman, Benjamin Wellner 
Information Technology Center, the MITRE Corporation 
Bedford, MA 
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THE GRAND CHALLENGE 


The project described in this presentation is part of MITRE’s larger ongoing 
research program in natural language processing (NLP) technologies, to include 
information extraction, information retrieval, question answering, speech, and dialogue. 
For the past several years, a handful of us have been working on developing a research 
paradigm in which we evaluate a computer’s understanding of language using the same 
tools and techniques used to evaluate a human student’s understanding of language. We 
have divided this project into three parts: Reading, Learning, and Teaching. 


The Grand Challenge 


Can we define a new research paradigm for natural language 
processing (NLP) by testing systems with measures for 
human comprehension? 

• Reading: Build a natural language processing system that 
can pass a reading comprehension test. 

• Learning: Create a system that improves its performance over 
time, based on understanding what it has already read (new 
words, concepts, facts). 

• Teaching: Create a reading companion that interacts with a 
student so that both system and student benefit. 


MITRE 
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VISION OF THE FRAMEWORK 


We are researchers in computational linguistics with an interest in education, 
rather than the other way around. We have thus been focusing our attention on 
developing a computer system that takes reading comprehension tests. But we are also 
interested in exploring how our system could perform as a peer learner in a larger 
teaching system. We have developed a primitive version of such a teaching system in 
order illustrate the interactions we envision. This teaching system will be described in 
greater detail on Figure 18. We would be very interesting in collaborating with education 
and learning technology researchers who could incorporate our reading comprehension 
system into a learning environment and test it with human students. 


Vision of the Framework 
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OUTLINE 


This talk will first present more information about our current research program, 
then describe two of the reading comprehension systems we have developed. The second 
part of the talk will present the teaching environments explored under this research 
program. 


Outline 


• Introduction to Research Program 

- Resources 

- Activities 

- Related Efforts 

• Reading Comprehension Systems 

- “Deep Read” 

- “ABCs” 

• Teaching Systems 

- Theoretical Foundations 

- Learning Interface 

- Authoring Tool 

- Interactive Learning Dialogue 
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EXAMPLE READING COMPREHENSION TEST 


Here is an example of one of the tests we use to develop and evaluate our reading 
comprehension system. It has a format familiar to everyone: a brief story plus a number 
of questions about the story. In order to make the research problem tractable, we have 
been using fact-seeking questions like the ones illustrated here. Reading comprehension 
tests also typically have questions such as “What is the author’s position on this issue?” 
While we recognize that these questions are ideal for encouraging critical thinking in 
human students, they are extremely difficult for computer systems, so our current 
research has not included them. Note also that the questions are short-answer type, not 
multiple choice as one would expect in a standardized test. We chose short- answer 
questions to make the research task more realistic; Life, after all, does not generally give 
one choices. We wanted to build a computer system that could answer the questions, not 
merely select the correct one from a set of options. 



Eugene Shoemaker may finally fulfill his lifelong wish to go to the 
moon. If all goes as planned on Saturday, a lipstick-sized container will 
carry his ashes to the moon's south pole. His ashes will be aboard 
NASA's Lunar Prospector. 

The Lunar Prospector is a moon-orbiting craft. It is expected to crash 
land on the moon early Saturday morning. NASA is hopeful that the 
impact of Prospector's crash into the moon will create a plume of water 
which will be visible from Earth with a telescope. NASA scientists are 
hopeful that this experiment will help them learn about the moon's water 
reserves. 

Shoemaker was a well-known astronomer and geologist. He was 
considered an authority on craters and the collisions that cause them. He 
is credited with discovering about 20 comets and 800 asteroids. 
Shoemaker gained worldwide fame in 1994, when he helped discover 
the comet Shoemaker-Levy 9. The comet slammed into Jupiter in 1994. 

It was always Shoemaker's dream to go to the moon. He applied to be an 
astronaut in the 1960s, but was rejected because of a medical problem. 
He later worked on several U.S. space missions, including the Apollo 
missions to the moon. He taught Apollo astronauts about craters. 
Shoemaker said shortly before his death the biggest disappointment in 
his life was “not going to the moon and banging on it with my own 
hammer.” 

© Canadian Broadcast Corporation, 1999 


Who learned all about craters from 
Shoemaker? 
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WHY READING COMPREHENSION TESTS? 


There are a number of reasons why we are using reading comprehension tests to 
drive our research. One is that they provide a means to measure the system’s capabilities 
and report these measures in a way everyone can understand. Another is that by basing 
the research on collections of genuine reading material, the results of the research will 
more likely transfer to real-world applications. We also feel that the task of taking 
reading comprehension tests moves information technology towards deeper language 
understanding. The task requires the computer system to understand not just the 
meanings of words, but how they relate to each other in the sentence (who is doing what), 
to understand how sentences relate to each other in the text, and the narrative 
organization of a text, such as temporal ordering of events. The questions are also 
completely open-ended, providing a true test of language skills rather than a test of 
specific domain knowledge or component skills such as name identification. 


Why Reading Comprehension Tests? 


• Provides an intuitive evaluation metric. 

• Grounded in genuine reading material (corpora). 

• Moves information technology towards language 

understanding: 

- Entire texts: 

• How words relate to each other within a sentence (e.g., 
who’s doing what) 

• How the sentences relate to each other (e.g., adding 
information, contradicting, changing topic) 

• The organization or narrative flow of the document 

- Open-ended: questions can be about anything. 
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DATA-DRIVEN RESEARCH PARADIGM 


The previous Figure mentioned the corpus-based nature of this research. This 
Figure explains in more detail how this type of research is carried out. We take a corpus 
(collection of texts) and divide it into three sets: training, development, and test. The 
training data is completely open to all the researchers on the project; it can be read and 
analyzed for patterns and content, or processed by machine learning algorithms. These 
activities result in the algorithms that perform the task — in this case, taking a reading 
comprehension test. The development data is closed to the system developers. They do 
not look at it, but they can test the system’s performance on this data and examine the 
results — a type of “practice test,” in other words. This process leads to further 
refinement of the algorithms. When the researchers are finally ready to see how the 
system is performing, they process the test data, which is data that neither the system nor 
its developers have ever encountered. 



Figure 7 
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MITRE CORPORA FOR TRAINING AND EVALUATION 


During the course of the Reading Comprehension research program we have 
created two separate data sets for developing and evaluating our systems. They both use 
texts based on a news story format. The Remedia Corpus consists of study materials for 
grade school children, purchased in hard copy form from Remedia Publications. The 
collection is divided into four sets of roughly 30 stories, with each set targeted to a 
different reading grade level, beginning at level 2. Accompanying each story are five 
short questions: who, what, when, where, and why. Each story is approximately 150-200 
words in length. The CBC Corpus was compiled based on material from the Canadian 
Broadcast Corporation’s CBC-4-Kids website. Each weekday this site posts a news story 
for young adults. Our corpus consists of 249 stories from the years 1999 and early 2000. 
We had professionally trained test item writers create eight to twelve questions for each 
story. The example story on Figure 4 comes from the CBC Corpus. In addition to the 
stories and questions, both corpora have undergone several types of linguistic analyses, 
and these analyses have been encoded into the text in the form of annotations. These 
include identifying names of certain entities and marking co reference, i.e., when a set of 
words each refers to the same entity. 


MITRE Corpora for Training and Evaluation 


• News story format 

• Remedia Corpus 

- Study materials for grade school children, purchased in 
hard copy format from Remedia Publications. 

- Four sets of 30 stories, 150-200 words in length, each set 
targeted to a different reading grade level, beginning at 
grade 2. 

- who, what, when, where, and why questions, plus an 
answer key 

• Canadian Broadcast Corporation (CBC) Corpus 

- 249 news stories for young adults, to which we added 
eight to twelve questions per story, plus an answer key 

• Annotation: Names of People, Places, Organizations; Times 

and Money; Co-referring expressions (Jackie the pilot 
she) 
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WORKSHOPS 


Two workshops have been held over the course of this research program. The 
first one was held at ANLP-NAACL in May of 2000. The Remedia Corpus was 
distributed to participants ahead of time so that they could each develop and evaluate 
their own reading comprehension system. Several college-level courses in Natural 
Language Processing used this task as a class project. During the workshop, participants 
reported on their systems’ design and performance. The second workshop was of a 
different nature. During this six-week workshop, a collection of researchers from 
different institutions came together at Johns Hopkins University during the summer of 
2000 to build a reading comprehension system. The CBC Corpus discussed earlier was 
created for this workshop, and the resulting system was called “Spot,” who will make an 
appearance in a later Figure on our teaching environment. 


Workshops 


• Workshop on Reading Comprehension Tests as Evaluation 
for Computer-based Language Understanding Systems 

- May 2000 at ANLP-NAACL* 

- Remedia corpus distributed beforehand; papers reported 
results 

• Reading Comprehension Workshop 

- Hosted by Johns Hopkins University under NSF funding 
(www.clsp.jhu.edu/ws2000/groups/reading/), Summer 
2000 (six weeks) 

- Used the CBC Corpus 

- Resulted in system called “Spot” 


* Applied Natural Language Processing / North American 
Chapter of Association for Computational Linguistics 
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HISTORY AND RELATED EFFORTS 


While space does not permit a full discussion of all the previous and related work 
in this area, we do want to mention two efforts that have particular relevance to learning 
technology. Wendy Lehnert’s work deserves special mention because she was one of the 
first to conduct research in the area of story understanding and question answering based 
on the story. Her system used sophisticated knowledge representation, reasoning, and 
inference. While the level of detail that needs to be encoded under this approach can be 
prohibitive in an open-domain real-world application, it may have applicability in a 
closed-domain learning scenario. An effort currently underway of particular note is 
NIST’s Question Answering evaluation, which has occurred for the last four years as part 
of the TREC evaluation. The systems are given a 3gb collection of text and as set of 
short fact-seeking questions to answer. One central obstacle inherent in this task is that 
human language allows for many different ways to say the same thing, as shown in the 
example below. 

(Discussion continued on next page.) 


History and Related Efforts 


• Wendy Lehnert (1981): Story understanding and question 
answering through sophisticated knowledge representation, 
reasoning, and inferential processing. 

- Requires extensive prior encoding of world knowledge. 

• NIST’s Text Retrieval Evaluation Conference (TREC) Question 
Answering Track 

- Given a question and collection of text (3gb), find the 
answer 

- Problem: many different ways to say the same thing 

What's the farthest planet from the sun? 

Pluto, the farthest planet from the sun in 
the solar system . . . 

Pluto usually is thought of as the ninth 
planet from the sun . . . 
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HISTORY AND RELATED EFFORTS (CONT.) 


To cope with this complex mapping of form to meaning, many system developers 
have chosen to utilize the WWW. By providing even more ways of saying the same 
thing, the WWW gives systems a better chance of finding the answer wording that their 
system is best able to associate with the question. Once they know what they are looking 
for, the systems return to the 3gb test corpus to find the answer. The highest ranking 
system that used this approach got 58% of the 500 questions correct. These results are 
impressive, but this approach would not be applicable to the reading comprehension task, 
in which the answer must be drawn from the story at hand. 

Fortunately, the most successful approach, taken by LCC, is very relevant to the 
reading comprehension approach. Their system did not use the WWW as a 
supplementary source of answers, and instead used lexical knowledge and logical 
inference — both ways of addressing the complex mapping of form to meaning. With this 
approach, 83% of the questions were answered correctly. We at MITRE are encouraged 
by these results because our current reading comprehension system is built along the 
same design philosophy. This system, called ABCs, will be discussed below. 


History and Related Efforts (cont.) 


- One approach: Use the WWW 

• Scored 58% correct and below. 

• Won’t work for Reading Comprehension task 

- Limited to given text passage 

- Text can be fiction 

- Most successful approach: Use lexical knowledge and 
logical inference 

• Moldovan et al. of Language Computer Corporation 
(LCC) 

• Scored 83% correct 
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MITRE’S FIRST READING SYSTEM: DEEP READ 


We have built a number of systems over the course of this research program. In 
this presentation we will show you two of them, the first one and the most recent. The 
first system, called “Deep Read” (pronounced Deep Red), is also known as the Bag-of- 
Words approach. It takes the words of each reading comprehension question, identifies 
the word stems and names of people, places, and organizations, and then puts the words 
into an unordered set. Next, it identifies the Focus of the question. In the example shown, 
the Focus is when a particular even occurred, so the system knows it’s looking for a 
temporal expression. Next, the system finds sentences in the story whose words overlap 
with the words in the question “bag.” The story has also been processed so that certain 
expressions like names and dates are flagged; these are shown in blue in the example. 
The candidate answer sentences are ranked according to which ones have the most word 
overlap (shown in bold type), and the system then outputs the sentence it thinks contains 
the answer. (Note that this early system did not return exact answers, but only sentences 
containing the answer.) 



Figure 12 
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DEEP READ RESULTS ON THE REMEDIA CORPUS 


We evaluated Deep Read’s performance using the Remedia Corpus. Overall, the 
system got 34% of the question correct. It performed best on who, what, and when 
questions, and less well on where and why questions. We were encouraged by the results 
of this rudimentary system, one which we built in order to obtain a baseline score. The 
results indicate that the problem is not trivial, but is tractable. 


Deep Read Results* on Remedia Corpus 


Percent Correct Sentences 



who what when where why overall 


• Automatically evaluated against an annotated corpus. 

• Conclusion: problem is challenging but tractable. 


* From Hirschman et al. ACL’ 99 MITRE 
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MITRE’S LATEST READING SYSTEM: ABCS 


We have taken a very different approach in our most recent system, called the 
Abduction Based Comprehension system, or ABCs. In creating this system, we had two 
goals. First, we want to always understand why the system succeeds or fails — to know 
what’s missing from its design. Second, as we move toward the teaching phase of our 
research, we want a system that is capable of interacting with a human learner in a 
meaningful fashion. ABCs thus has a number of characteristics designed to meet these 
goals. It contains discrete components that can be evaluated independently. It returns 
exact answer phrases rather than answer sentences and it exploits knowledge sources and 
even adds to them based on information encountered in the story. ABCs most powerful 
feature is the logical inference engine, which the system uses to locate and justify 
answers to the questions. The inference engine gives us access to the system’s “thought 
process,” making it more suitable for interacting with a human learner. 


MITRE’s Latest Reading System: ABCs 


Abduction Based Comprehension system 

Goals: 

• Understand how and why the system succeeds or fails. 

• Create a system capable of participating in a learning and 
teaching environment. 

Characteristics: 

• Returns exact phrase answers. 

• Contains discrete components that can be evaluated 
independently. 

• Exploits knowledge sources, and adds to them. 

• Uses a logical inference engine, which allows the system to 
justify its answers (via a logical proof). 

• More natural, intuitive access to the system’s “thought 
processes” makes it better suited for interacting with users. 
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ABCS ARCHITECTURE 


The example shown here illustrates the information flow through the system. 
ABCs takes the entire story and sends it through a number of language processors. These 
processors perform linguistic analyses on the text, including name identification, sentence 
structure (subject, object, predicate), normalization of temporal expressions, and 
identification of expressions that refer to the same entity (“co reference”). The questions 
are processed in the same way. Next, this output goes through a semantic interpretation 
process to create a set of facts that are added to the Knowledge Base, which also contains 
existing knowledge in the form of a lexicon. ABCs uses the lexicon to identify word 
meaning as well as synonyms. The questions undergo a similar process, where they are 
turned into formal queries with a variable assigned to phrases in the question. The 
Abductive Proof Procedure then attempts to match phrases in the Knowledge Base to 
variables in the Query, using abduction where needed. From this, one or more candidate 
answers are generated. In some cases, the Answer Generator then chooses the best way 
to present the answer. For example, if the question in this case were “Who helped 
discover the comet. . the system would first find “he” as the answer since it is the local 
subject of the verb “helped” in the story. The Answer Generator would look up who “he” 
refers to, and output “Eugene Shoemaker” as the answer. 


ABCs Architecture 


Story 

. . . Shoemaker gained worldwide 
fame in 1994, when he helped 
discover the comet Shoemaker- 
Levy 9. ... 


Story 

Semantic 

Interpretation 


k 


Language 

Processors 


\( 


<person>Shoemaker</> gained worldwide 
fame in <val= 1 994> 1 994</>, when he helped 
discover the comet Shoemaker-I^M pl . S 

obj s - ^ V_^co-refer 

When did <person>Shoemaker</> help 
discover Shoemaker-Levy 9 ? 




Question L 
Semantic 
Interpretation 

Query 

• • • • • • 

1 

f 

Question 

When did Shoemaker help 
discover Shoemaker-Levy 9? 


1994 

|person| |event| |temporal| 




1 1 1 
he help when 
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ABCS RESULTS ON THE REMEDIA CORPUS 


We recently evaluated the performance of ABCs on the Remedia Corpus. The 
chart shown here shows the results for two sets data sets: Development and Evaluation 
(see Figure 6 for a discussion of the different data sets). Although the system performed 
consistently better on the Development data, the difference is not statistically significant 
due to the small size of the data sets (30 stories for Development, and 60 stories for 
Evaluation). Note that these results are not directly comparable to those of Deep Read, 
because ABCs has not yet been evaluated on why questions and ABCs also have the 
harder task of returning exact answers. These results were obtained by having a human 
grader evaluate the system’s response, using an answer key as a guide. An answer was 
deemed inexact rather than correct if the answer was not worded properly; e.g., the 
system returned “the first woman” instead of “the first woman to fly faster than the speed 
of sound.” Overall, the system answered around 45% of the questions correctly, 
performing quite well on who and when questions. What questions are particularly 
difficult; one reason for this is that while the Focus of a who question is fairly easy to 
interpret, the Focus of what questions is much more variable. In the near future we plan 
to build additional components for ABCs, enhance its logical inference process, and 
evaluate it with additional corpora such as the CBC Corpus. 
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MITRE’S TEACHING SYSTEM: THEORETICAL FOUNDATIONS 


In the second part of this presentation we will discuss the learning environments 
that could incorporate the reading comprehension systems described earlier. In designing 
prototype learning environments, we have adopted a technique known as Reciprocal 
Teaching. Originally designed to teach reading skills to humans, it takes the form of a 
dialogue about a text between a teacher and a group of students. Everyone takes a turn at 
leading the dialogue. As part of the dialogue, participants ask each other reading 
comprehension question as well as clarification questions. A large collection empirical 
research has demonstrated how effective this technique is both for reading skills and 
other educational domains. Cognitive Apprenticeship is another philosophy that we’ve 
embraced while developing ideas for a reading comprehension tutor. Given this 
background, our research question has become, “Can a system like ABCs act as a 
simulated peer learner in a Reciprocal Teaching dialogue?” 


MITRE’s Teaching System: 
Theoretical Foundations 


• Reciprocal Teaching (Palinscar & Brown, 1984), a technique 
for teaching reading skills to humans: 

- Takes the form of a dialogue about the text between a 
teacher and a group of students. 

- Everyone takes a turn leading the dialogue 

- The dialogue is structured around four tasks: 

• Question Generating •Summarizing 

• Clarifying • Predicting 

- 26 published studies with human subjects demonstrate its 
effectiveness (Rosenshine et al, 1996) 

• Cognitive Apprenticeship (Collins, Brown, & Newman, 1989) 

- Externalizing cognitive processes so students can 
observe and learn the processes involved. 
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CAN THE COMPUTER BE A PEER LEANER? 


The diagram shown here sketches how a dialogue might play out between a 
simulated computer peer and a human student who is leading the dialogue. The human 
student would first ask a question. Ideally, a subcomponent would evaluate the quality of 
the students question and guide her in asking questions that will enhance her own 
understanding of the reading material. The system could then either posit an answer or 
ask a clarification question. The human student would evaluate the correctness of the 
system’s answer or respond to the system’s clarification question. If the human student 
indicates that the system’s answer is wrong, the system can try again, ask a clarification 
question, or just give up and ask for the answer. Regardless of the outcome, the system 
benefits by logging the exchange for future training. Under the theory of Reciprocal 
Teaching, the human student benefits by asking questions about the text — a very useful 
comprehension strategy. 


Can the Computer be a Peer Learner? 


Human Student reads a portion of 
text and poses a QUESTION. 





Human 

teacher or 

artificial 

teacher 

evaluates 

quality of 

question. 


Human Student 
agrees. 


Artificial Student registers 
interaction as positive training 
example for learning 
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RECIPROCAL TEACHING ENVIRONMENT 


We have developed a rudimentary graphical user interface to explore this idea 
further, and to give us a means to demonstrate to potential collaborators the type of 
environment we envision. In the left half of the screen is the story that is being read, and 
in the right half is the history of the dialogue about the story. There are three participants 
in the dialogue. One is a simulated teacher whose main function is to facilitate the 
exchange between the other two participants, and also to ask questions of the human 
student, since our reading comprehension system does not ask questions. The second 
dialogue participant is the human student, and the third is one of our Reading 
Comprehension systems, in this case, Spot, mentioned earlier in Figure 8. The first five 
lines in the dialogue show the human student answering a question presented by the 
teacher. The teacher’s questions come from a stored list of questions. The student’s 
responses are being “graded” by a separate component that compares the answers against 
an existing answer key. The final four lines of the dialogue show Spot taking a turn at 
answering the human student’s question, which the human then evaluates as correct. 
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AUTHORING TOOL 


As the previous Figure demonstrated, much of the material in this interface can be 
created in advance by professional courseware developers. To facilitate the creation of 
stories, questions, and answer keys, an authoring tool would be an essential component of 
this learning environment. This would give instructors the ability to develop their own 
materials while at the same time eliminating the need for them to modify the natural 
language processing technology underlying the software. 


Authoring Tool 


• Provides instructional 
and domain flexibility. 

• Human instructors can 
create the questions 
and specify the 
acceptable answers for 
each passage. 

• Makes the natural 
language technology 
accessible to 
educators. 


create a new story, fill i 
Story title: 
Story Date: 
Source Document: 
Copyright Info: 
Language: 


;e optional fields and press OK 


CBC 


nEnglishJ 

Cancel | Help | 

1 15? |Unsigned Java Applet Window 


I N Reading Comprehension Authoring Tool 


File Edit Help 

Title: Last Eclipse of the Millennium 
Passages in the current story: 



Answers to the current question: 


Remove selected answer from the question | 


Enter text of new passage: 


A total eclipse is one of the most 
impressive natural occurrences. 

It is caused when the moon passes 
between the Earth and the sun, blocking 
our view of the sun for a few minutes. 


4 


Questions in the current passage: 


What passes between the Earth and the 


I A 

Remove selected question from the passage 



A 

Attach this passage to the story | 

edit | 

Enter text of new question: 

clear | 

What passes between the Earth and the 1 

sun during a Solar Eclipse? 

A 

Attach this question to the passage | edit | 

Enter text of new answer: 

clear | 

the moon 

A 

A 

Attach this answer to the question 

1 edit | 


I jUnsigned Java Applet Window 
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CLARIFICATION QUESTIONS FROM ABCS 


Earlier we saw Spot answering fact-seeking questions. Another process that 
Reciprocal Teaching encourages is the asking of clarification questions. Our new ABCs 
system, with its use of logical reasoning, is able to do just that. In the example below, the 
story says early on, “Six sailors finished a 5,000-mile sailboat trip yesterday.” Later, it 
states, “The name of their boat is the Kon Tiki.” When asked “What is the name of their 
sailboat?” the system responds with a clarification question “Is [their, boat] an instance of 
a sailboat?” It is trying to abduce that “their boat” is a sailboat, since the question 
mentions “their sailboat.” Once the user confirms the validity of this abduction, the 
system posits the answer, “Kon Tiki.” This interface was only a couple of weeks old at 
the time of this writing, and much more would need to be done to make the clarification 
questions easier to understand, and determine what type of clarification questions would 
be beneficial for the human student to encounter. Nevertheless, this demonstrates how 
reasoning-based systems can add a degree of naturalness to simulated peers. 


Clarification Questions from ABCs 


Stoiy 


1989 Remedia Publications, Comprehension/5W 
Sailors End Long T rip(SOUTH S EA ISLANDS, August 1, 1947) Six sailors 
finished a 5,000-mil^ailboat trij^esterday. The men crossed the sea just 
like people did long ago. Their boat did not have an engine. All they had were 
oars. The men left Peru on April 28. They wanted to show how, in the past, 
people traveled miles across the seas. For supplies, they took food and water. 
They cooked fish on a small stove. For company, they took along a parrot. 

The captain's name is Thor. Since he landed, he has told many tales. Often, 
he and his crew ran into storms. One time, someone fell overboar d. They 
quickly saved him with ropes. For months they did not see land. At last, they 
spotted some birds. But more weeks went by without land. On day 101 of the 
trin. they reached the shores of the South Sea Island&^ The name of their boat) 
is the Kon Tiki. It travslfid mill 1 ' Fill Mil 1 IHI'li lllHi'l pi ill In nl il Back „ 


Choose an action: 


Open New Story 


Query Analysis 


Dialog Session 

What is the 
name of the sailboat? 


ieii 


i m->ls 

[their, boat ] an instance 
of a sailboat? 

& 

yes 


*rx->is 

[ the, Kon, Tiki ] the 
correct answer? 


Your input: Virtual Keyboard 

yes 
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SUMMARY 


Natural Language Processing systems are not perfect, and won’t be for some time. 
But there are certain scenarios and teaching philosophies in which the computer does not 
have to be perfect in order to be a useful participant in a learning environment. In 
particular, the reading and question-answering component can play the role of a 
simulated peer, since real peers are not perfect either. This scenario is further enhanced 
with reasoning-based systems, which can interact with human students in a dynamic 
dialogue. 


Summary 


• The computer doesn’t have to be perfect to be a good 
pedagogical tool. 

• The computer can play many roles with different components: 

- Teacher 

- Student 

- Grader 

• A computer system capable of reasoning and learning can act 
as a student peer in an interactive learning environment: 

- Asking (and knowing when to ask) clarification questions 
(e.g., about word meaning) and then remembering and 
using the feedback to improve performance. 

- Providing justifications for answers. 
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A SYSTEMATIC APPROACH TO RESEARCH: 
ROADMAPPING ACTIVITIES BY THE LEARNING FEDERATION 


This talk will discuss why it is necessary to conduct an inter-industry 
roadmapping exercise to encourage greater innovation in educational technology 
development. 


A Systematic Approach to Research: 
Roadmapping Activities by 
the Learning Federation 

Marianne Bakia 

Federation of American Scientists 
April 3, 2003 


Figure 1 


Overview of My Talk 

□ Needs 

□ Opportunities 

□ Barriers 

□ Intro to the Learning Federation 

□ Systems Engineering Approach 

□ Roadmapping Process 

□ Next Steps 
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THE NEED 


The need to develop more powerful educational technologies is based on a unique 
confluence of events that include an increase in the number and diversity of students 
seeking education, the In addition to serving the new, enlarged student body, education 
systems are also being asked to emphasize reasoning, problem solving, effective 
communication and collaboration at all levels of education. These new demands come at 
a time when education budgets are under tremendous pressure. 


The Need 

□ Americans are entering education and training 
institutions in much greater numbers, with increasing 
frequency, and with greater diversity in terms of age, 
experience, background and need. 

□ Workers at all levels need higher levels of knowledge 
and skills, including reasoning, problem solving, 
effective communication and collaboration. 

□ Employers increasingly need education and training for 
their workers on a “just in time” basis. 

□ Schools typically operate on severely constrained 
budgets. 


Figure 3 
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THE OPPORTUNITY 


There is a wide variety of software-based tools and services being developed for 
education. These tools facilitate every aspect of the educational process including 
teaching, learning, and administration. Common applications include a phenomenal 
increase in access to scholarly and informational sources over the web through digital 
libraries and related collections. Technology can also improve the teaching of complex 
knowledge and skills through visualizations, models and simulations. Teaching tools like 
“intelligent tutoring systems” are also changing teaching norms and improving learning 
outcomes. Through distributed learning and virtual worlds, students often become more 
motivated and can apply new knowledge and skills in powerful ways. Learning 
management tools are also emerging that can lighten the burden placed on teachers to 
individualize instruction and keep track of administrative information. 


The Opportunity 

□ Large Scale Digital Libraries and Online 
Museums 

□ Visualization, Modeling, and Simulation 

□ Intelligent Tutoring Systems 

□ Distributed Learning and Collaboration 

□ Virtual worlds 

□ Learning management tools 
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BARRIERS 


These applications are still in an early stage of development. Research is needed 
in an array of areas related to information technology, software design, and the processes 
of cognition, learning, and memory. Developing an instructional technology may require 
multidisciplinary effort — both in terms of research and product development 

Unfortunately, there are few bridges between cognitive researchers, commercial 
producers, inventors of learning technologies and commercial product developers, and 
practitioners. Individual developers are often forced to design every aspect of a system 
with inadequate resources. 


Barriers 


□ Research needed in an array of areas related to 
information technology, software design, and the 
process of cognition, learning, and memory. 

□ Developing an instructional technology may 
require multidisciplinary effort — both in terms of 
research and product development. 

□ Few bridges between cognitive researchers and 
commercial producers; inventors of learning 
technologies and commercial product 
developers; and practitioners (and among 
producers with common generic technology 
needs) 
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A DRAMATICALLY DIFFERENT APPROACH TO RESEARCH AND 
DEVELOPMENT IS REQUIRED: THE LEARNING FEDERATION 


If we are to realize the power of technologies to improve education, a 
dramatically different approach to research and development is required. The Learning 
Federation has been organized to facilitate this transformation. We are working to 
catalyze a revolution in learning. 


A Dramatically Different Approach 
to Research and Development is 
Required: 

The Learning Federation 

Working to Catalyze a Revolution 
in Learning 
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THE LEARNING FEDERATION 


The Learning Federation began with the creation of a SEMATECH-like 
consortium that includes industry, academia, government, foundations. The first phase of 
operation includes the development of a research roadmap that includes key priorities for 
future innovation and details a plan of action to accomplish these goals. The second 
phase of operation will launch a program that operates at between $100M-$200M/year a 
year to execute the roadmap. This program will support large-scale, sustained efforts 
involving multiple disciplines and drawing on expertise from academia, industry, and 
government laboratories. 


The Learning Federation 

□ Create a SEMATECH-like consortium: industry, 
academia, government, foundations 

□ Develop a research roadmap 

□ Fund ($100M-$200M/year) and execute the 
roadmap 

□ Support large-scale, sustained efforts involving 
multiple disciplines and drawing on expertise 
from academia, industry, and government 
laboratories 


Figure 7 
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LEADERSHIP 


The Learning Federation’s leadership includes representatives from America’s top 
universities, businesses, and government agencies. Our steering committee includes 
many leaders from computer science and cognitive science. Microsoft, Hewlett Packard, 
and the Corporation of Public Broadcasting are founding corporate members. 



Leadership 

Steering Committee 

Executive Management 

■ 

Ruzena Bajcsy, UC Berkeley 

Henry Kelly, Federation of American 

■ 

John Bransford, Vanderbilt U 

Scientists 

■ 

Randy Hinrichs, Microsoft Research 

Marianne Bakia, FAS 

■ 

Ed Lazowska, U Washington 

Kay Howell, FAS 

■ 

Elliott Masie, Masie Center 

Tom Kalil, UC Berkeley 

■ 

Richard Newton, UC Berkeley 


■ 

Don Norman, Nielsen Norman Group 


■ 

Raj Reddy, CMU 


■ 

Shankar Sastry, UC Berkeley 


■ 

Bill Spencer, Washington Advisory Group 

■ 

Janos Sztipanovits, DARPA, Vanderbilt U 

■ 

Andries van Dam, Brown U 


■ 

Karen Watson, CPB 


■ 

Ann Wittbrodt, HP 



Figure 8 
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LF MISSION 


The mission of the Learning Federation is to dramatically increase the funding 
available for the research and development of educational technologies. We are working 
closely with the Digital Promise Project (www.digitalpromise.org) to establish a National 
trust fund for innovation in educational technologies. These funds will be used to 
encourage a systematic, collaborative approach to research and development. The focus 
of research and development will be on pre-competitive tool development for next 
generation technologies. 


LF Mission 

□ To dramatically increase funding for 
research and development of educational 
technologies (Digital Promise) 

□ Encourage a systematic, collaborative 
approach to research and development 

□ Focus on pre-competitive R&D for next 
generation technologies 


Figure 9 
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SYSTEMS ENGINEERING 


The Learning Federation’s efforts are designed to launch education technology 
development out of the “cottage industry style” into an industrial mode that allows for 
interoperable components to be individually developed and sold. The design of 
environments will be guided by the latest in cognitive research and scientifically-based 
pedagogical approaches. Tools sets will expand the development of simulations and 
immersive environments that will help them practice actions and procedures as easily as 
they can study facts and concepts. Learner modeling and assessment tools will validate 
teaching processes and help ensure that curriculum is tailored to individual needs. 
Research shows that feedback and interactivity are important components of any learning 
system, and tools for question generation and answering as well as more general 
communications technologies will facilitate these aspects of learning. Integration tools 
will facilitate the easy connection of the components into specialized learning systems. 



Figure 10 
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THE LEARNING FEDERATION 
LEARNING SCIENCE AND TECHNOLOGY R & D ROADMAP 
WORKSHOPS PROJECTS 


This figure details the components of the roadmap. A workshop of 40 national 
experts in relevant areas has been developed for each topic. The instructional design 
workshop includes cognitive scientists, educational technology developers, and 
instructional designers. The question generation and answering workshop brought 
together help-desk operators, linguists, and educational technology developers to 
highlight best practice and identify continued research needs. 


The Learning Federation 

Learning Science and Technology R&D Roadmap 
Workshops Projects 


Instructional Design for New Technology-Enabled Approaches to Learning (Simulation-Based Learning Systems for | 
STEM - Issues in Design and Assessment): This workshop will examine how to build on emerging information about the 
biological basis of learning to design effective technology-enabled learning environments. The workshop will examine current use 
of simulation-based education and training and identify research priorities for designing and evaluating the use of advanced 
simulation capabilities in future learning systems. Key topics for discussion include: 

■ Developing multi-dimensional models of subject-matter mastery and expertise in different subject areas 

■ Understanding the influence of variables that may affect learning: motivation, prior experience, interest, learning styles 

■ Developing tailored approaches based on prior experience, learning style, etc. 

■ Determining when and how to use discovery-based, game-based learning, and other approaches enabled by new technology, | 
relative to traditional practices such as lecture, reading print, or drill and practice 

■ Developing principles for instructional design: when to use group vs. individual learning, media use 

Determining optimal roles for physical presence/human intervention 

■Question Generation and Answering Systems for Technology-Enabled Learning Systems: This workshop will examine 

■ how to apply what we know about question generation and question answering to design learning environments that guide learners | 
Ito actively construct knowledge. The workshop agenda includes R&D to enable responses to learner inquiries through automated 

■ responses and dispatch of questions to instructors and experts, and methods for increasing the frequency and quality of questions. 

| Key topics include: 

Assisting learners to formulate answerable questions 
Helping learners identify their own knowledge deficits to stimulate questions 

Diagnosing sources of misunderstanding and proposing new directions (with or without instructor assistance) 

Using question sessions to build profiles of the learner's interests, capabilities, and learning styles in order to make the 
system more effective in responding to the learner's needs 
Natural language dialogues 

Building technical systems to identify which source to use to answer the question - FAQs, multimedia libraries, instructional 
simulations, other web sources, instructors, experts, peers. 

Qualifying the sources in terms of relevance and accuracy 

Dispatching the questions to the answer resource in such a way that the answer resource can answer it correctly 


Figure 11 
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THE LEARNING FEDERATION 
LEARNING SCIENCE AND TECHNOLOGY R & D ROADMAP 
WORKSHOPS PROJECTS 


With the current emphasis on testing and accountability in education, new 
technologies could help create affordable, more robust student assessments. These 
assessments are also a necessary element of individualizing curriculum to student needs 
and interests. By monitoring motivation and comprehension, these tools could capture 
some of the most promising techniques of human tutors, which are too expensive for 
massive deployment in all educational environments. With key component tools 
developed, it will also be necessary to have integration tools that can link each of the 
individual elements into a comprehensive system. 


The Learning Federation 
Learning Science and Technology R&D 
Roadmap Workshops Projects 


Assessment Tools for Technology-Enabled Learning Systems - What to Measure, When to Measure, How to Use 

the Information: The workshop will study how to apply learning science to develop multi-dimensional measures of 
learners' approaches to expertise in SMET areas of interest. The workshop will identify research priorities for designing 
and building tools to gather information for assessment, inform software and instructors for efficient management of the 
learning process, and provide iterative feedback loops. Key topics include: 

■ Measuring individual and group skills 

■ Measuring levels of learner interest and motivation in complex environments (including discovery-based learning 
scenarios) 

■ Defining useful measures of learning styles and measuring how they may be revealed by student performance 

■ Embedded, continuous, multi-dimensional assessments of content mastery; embedded systems to discover 
individual learning styles. 

■ Programmable tools that use the information gathered to trigger appropriate automated responses in the 
courseware (active course management) or set alarms to summon teachers/experts 

■ Ensuring security and privacy of assessment information. 

■ Management and record keeping including public and private portfolios (available only to learners and their 
teachers) 

Integration Tools for Content and Instructional Design of Technology-Enabled Learning Systems: The 

workshop will focus on R&D priorities for building tools to translate learning theories into functional learning environments 
and establish an open process for worldwide collaboration on building and maintaining learning environments. The 
workshop agenda will focus on: 

■ Developing course building tools for designing scenarios, creating assignments, designing response to information 
gathered from student observer tools, and programming avatar behaviors 

■ Designing tools to allow for easy formation of teams/groups even when members are geographically dispersed 
(P2P tools, building and managing networks of learners, students, and specialists.) 


Figure 12 
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THE ROADMAP PROCESS 


The research roadmap is the plan that will guide this development process. The 
roadmap is being developed in multiple stages. First, we organized a workshop of the 
nation’s leading experts in learning technologies to identify principal research areas. 
Next, we developed research priorities within research area that was identified. We 
developed a strawman outline with key headings, goals and research tasks, which was 
revised based on literature reviews, interviews with experts. In a series of workshops, we 
have solicited peer review of roadmap components. These components will be integrated 
into a final report. The report will be used to launch the necessary research. Because 
planning is a continuous process, we will be repeating the roadmapping process on a 
regular basis. 


The Roadmap Process 

1 . Identify principal research areas 

2. Develop research priorities within each area 

- Brainstorm to design a strawman roadmap 

- Undertake literature reviews, interviews, all means 
necessary to perfect and extend strawman 

- Peer review of proposed strawman 

3. Combine component roadmaps to a integrated 
roadmap 

4. Launch Research 

5. Repeat steps 1 -3 on a regular basis 


Figure 13 
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THE LEARNING FEDERATION ROADMAP WORKSHOPS PROJECTS 


The Learning Federation is in the process of completing the first phase of 
operation. Workshops have been completed, and we are in the process of finalizing 
component roadmaps. 


The Learning Federation Roadmap 
Workshops Projects 


I Workshop Description 

! Preparation Status 

Date/Location Workshops 1 

Question Generation and Answering 
Systems for Technology-Enabled Learning 
Systems 

- preliminary roadmap published 
and disseminated for review & 
comment 

- workshop to produce final 
roadmap announced and 
invitations sent 

Oct. 4 & 5 2002 at Univ. of Memphis to develop 
preliminary roadmap 

April 10 & 1 1 at MITRE Bedford, MA to 
develop final roadmap 

Instructional Design for New Technology- 
Enabled Approaches to Learning 

- Industry visits in progress 

- Draft roadmap in progress 

December 4 &5 2002 at Univ. Central Florida 
Simulation Center, Orlando FL 

Assessment Tools for Technology-Enabled 
Learning Systems 

- Planning in progress 

May 4 -6, 2003 Seattle, WA 

Virtual Environments to Support 
Exploration-Based Pedagogy 

• Next Generation Approaches for the 
Geometrical Modeling and Dynamic 
Simulation of Biological Systems 

- workshop report in progress 

Planning for Workshop op 
Navigation Tools in progress 

- Geometrical Modeling workshop held January 
25 2003 Newport Beach, CA (in conjunction with 
the Medicine Meets VR Conference) 

Integration Tools for Content and 
Instructional Design of Technology-Enabled 
Learning Systems 

- Focus topics identified 

TBD 

Integrated Roadmap 

-looming on the horizon 

July 2003 at IDA Washington, DC 


Figure 14 
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NEXT STEPS 


Once we have completed the roadmap in August, we will launch a public 
campaign to communicate the need for increased public investment through the fall of 
2003. This campaign will include constituent outreach with researchers, developers and 
practitioners. 


Next Steps 

□ Complete Workshop Series July 2003 

□ Complete Draft Roadmap August 2003 

□ Launch Public Campaign about Needs 
and Opportunities, September 2003 

□ Constituent Outreach Continuing through 
Dec. 2003 


Figure 15 


Please Stay Tuned! 


www.thelearningfederation.org 


Contact Information for Marianne Bakia 

mbakia@fas.org 

202-454-4689 


Figure 16 
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Intelligent Agents for Distance Learning 


Brad Goodman 
The MITRE Corporation 
Bedford, MA 


Reprinted with the permission of The Mitre Corporation. 
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INTELLIGENT AGENTS FOR DISTANCE LEARNING 


Our research project is exploring techniques for providing stimulating peer 
learning experiences in web-based, collaborative distance-learning environments. We are 
leveraging intelligent tutoring system and collaborative learning technologies to develop 
an artificial learning agent to collaborate with human peers. The agent will follow the 
discussion and interact with the participants when it detects learning trouble of some sort, 
such as confusion about the problem they are working on or a participant who is 
dominating the discussion or not interacting with the other participants. In order to 
recognize problems in the dialogue, we are first examining the role that a participant is 
playing as the dialogue progresses. This simulated peer will play supportive roles based 
on the instructional needs of the human learners. This talk describes our pursuit of 
indicators for effective and ineffective collaborative learning. Potential indicators such as 
communicative acts, participant roles, and other conversational elements will be 
discussed. This project builds on a collaborative-learning infrastructure developed in a 
previous MITRE research project to bridge distance and time barriers between on-line 
learners [8, 19, 21]. 
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OVERVIEW 


This talk will introduce the training issues that led up to this research. Background 
to motivate the research will be given. Research problems will be presented. The area of 
collaborative dialogue modeling will be introduced. A description of the analytical 
techniques employed in this research to predict when instructional intervention was 
required is presented. Lastly, a description of an intelligent agent peer that is under 
construction will be given. 


Overview 

■ Introduction 

■ Research Motivation 

■ Research Problems 

■ Collaborative Dialogue Modeling 

■ Protocol Analysis for Intervention Prediction 

■ Pierce, An Intelligent Agent Peer 


MITRE 

& 2WB, The M ITR E Co rporalion 
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PROBLEM 


Soldiers face conflicts in a rapidly changing world. These conflicts cannot be 
fully anticipated when training is developed for the schoolhouse. Therefore soldiers must 
adapt on the fly to ill-defined battlefield situations applying what they have learned & 
reasoning about the situation at hand. Relying on schoolhouse solutions won’t work. 

Classroom learning improves considerably when students participate in structured 
learning activities with small groups of peers [2]. Peers can encourage each other to 
reflect on what they are learning and to articulate their thinking, which enhances the 
learning process [5], Reflective activities, such as After Action Reviews in military 
exercises, encourage students to analyze their performance, contrast their actions to those 
of others, abstract the actions they used in similar situations, and compare their actions to 
those of novices and experts [5, 13]. 


Problem 


■ Continuously changing 
threats, procedures, and 
equipment 

-Invalidates current 
schoolhouse instructional 
paradigm 

-Train competencies 

-Requires continual 
education 

■ Movement to distance 
learning (anytime/anywhere) 

-Savings: TDY cost & time 

-Loss: Instructor & peer 
contact 



Interaction with peers is a fundamental component of 
successful training. 


MITRE 
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BACKGROUND 


Our work is an outgrowth of research in both the intelligent tutoring system and 
collaborative learning communities [13]. We first developed a simulated learning 
companion capable of acting as a peer in an intelligent tutoring system [4, 7], The 
presence of the learning companion can help ensure the availability of a capable 
collaborator to enrich learning through the promotion of reflection and articulation in the 
human student. In other work on distance learning, a student is allowed to work with 
students in other locations through a collaborative system [8, 13, 17], 

Our current work is attempting to combine these approaches by allowing students 
to work both with a collaborative system and with an electronic peer. The overall goal is 
to provide an electronic peer that can co-exist in cyberspace with human collaborators 
and interact as a partner to promote effective collaborative learning and problem solving. 
The peer monitors the discussion and interacts with the participants when it detects 
learning trouble of some sort, such as confusion about the problem they are working on, 
or a participant who is dominating the discussion or not interacting with others. 


Background 



LuCy: Hi! This is LuCy, your peer. 
I think selecting that evidence 
was a bad idea. Why did you do 
that? 

Sarah: I have seen data that 
supports this evidence. 

LuCy: What data do you think is 
related. Brad? 

Brad: (MINT Report that 
helicopters are flight-ready. 

LuCy: I agree. 

Sarah: Okay. 


MITRE 
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OBJECTIVE 


The current focus of our work is on automatically detecting such learning 
problems. The actions of participants, with respect to chat and workspace tools, provide 
an initial indicator of discussion and progress towards completing an assigned task or 
dealing with an obstacle. The roles of participants can provide a secondary indicator. We 
will discuss our use of participant roles as indicators of group activity, how we determine 
them statistically, and our plans for moving on to the next step. 


Objective 


■ To develop and incorporate a simulated peer 
into a rich , multi-person collaborative 

dis tance-learning en vironm en t to fa c i I i tate 
both group and individual learning 

■ To infuse and assess instructional roles in 
the simulated peer to promote effective 
learning 


MITRE 
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SET A BASELINE: COOPERATIVE PROBLEM SOLVING 


Our research in learning companions motivated us to investigate collaborative 
learning among students more closely. We studied the dynamics of collaborative learning 
groups [19, 21] by observing students working together to solve a common problem. The 
study was conducted during a five-day course in which students learned and used a 
software design methodology, Object Modeling Technique (OMT), to collaboratively 
design software systems. The students were videotaped as they worked in groups of four 
or five. The videotape transcriptions were coded with a conversational act-based coding 
scheme, and studied using summary and sequential analysis techniques. We found that 
students learning in small groups encourage each other to ask questions, explain and 
justify their opinions, articulate their reasoning, and elaborate and reflect upon their 
knowledge, thereby motivating and improving learning. 


Set a Baseline: Cooperative 
Problem Solving 


Videotaped students .collaboratively solving 
software system design problems 


Videotaped 
groups 
of 4 or 5 



© 

Coded 

their 

interactions 


© 

Analyzed 
the 
data 


Person 

Dialogue 

Learning Skill 

f 

StibskEII 

W1 

No, a corporation's made up 
of companies. 

Acknowledge 

Reject 

M3 

One on me there. A company 
has departments. 

Inform 

Assert 

W1 

We have to draw a picture, 
right? 

Task 

Coordinate 
arouo process 

M3 

Department, that takes the 
up arrow there, richt? 

Inform 

Assert 

W1 

Does it inherit, does 
department inherit the 
characteristics of the 
company? 1 don't think so, 

Inform 

Assert 


) 


Utilization of CL Skills 


mix 


i 


/ / 
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A FIRST STEP TOWARDS AN INTELLIGENT AGENT PEER 


We followed our initial study with the development of a collaborative learning 
environment for OMT. Our goal was to provide a web-based environment that would 
permit us to analyze peer-to-peer dialogue and tool actions in an attempt to identify the 
strength and weaknesses of a group’s interaction. As part of that study, we looked for 
indicators that the learning process was in trouble. 


A First Step Towards an 
Intelligent Peer Agent 

■ We need to be able to predict when the 
learning process is in trouble. 

- Mark: We need to add a discriminator. 

- John: What’s that? 

- Mary: No we don’t. 

- Mark: I’ll add one. 

■ This prediction must be done as the dialogue 
progresses so the intelligent peer agent can 
intervene 


MITRE 
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DRIVING AN INTELLIGENT AGENT 


The actions of an intelligent agent in a collaborative learning context could be 
governed by peer-like behavior, pedagogical behavior, and group dynamics. Peer-like 
behavior can be derived from traditional intelligent tutoring system research. Expert 
knowledge about the domain under instruction and typical student misconceptions can 
delineate the information the agent should convey. How that information is presented to 
the student is based on typical pedagogical behavior in a collaborative instructional 
situation. Instructional principles and available instructional resources will determine 
what behavior should be exhibited by the intelligent agent. Lastly, group dynamics plays 
an important role in determining how to structure the interaction between the agent and 
the students. The effectiveness of communication between the students can be examined 
and, where necessary, improved. The agent can assume a missing role to help improve 
group interaction. 


Driving an Intelligent Agent 

■ Peer-like behavior 

- Expert knowledge 

- Misconceptions 

■ Pedagogical behavior 

- Instructional principles 

- Instructional resources 

■ Group dynamics 

- Communicative effectiveness 

- Role playing 

MITRE 
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PLANNING AGENT ACTIONS 


The actions of the intelligent agent must be carefully planned to cover the issues 
outlined in the last slide. A planning component would need to answer a set of questions 
about the group interaction and the instructional needs to best help facilitate learning. 


Planning Agent Actions 



pe 




1 


W ^ oup< , ynami c,- ntefactionsareobs6 _ 


What roles are 


being played by participants 
Seal '«#*** 


Peer Agent 
Planner 
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GOAL: PROMOTE IMPROVED GROUP DYNAMICS FOR LEARNING 


Small group dynamics research can benefit collaborative learning. The promotion 
of more effective interaction in a group can increase the learning of the group and its 
members. Students learning in small groups encourage each other to ask questions, 
explain and justify their opinions, articulate their reasoning, and elaborate and reflect 
upon their knowledge, thereby motivating and improving learning. These benefits, 
however, are only achieved by active and well-functioning learning teams. Placing 
students in a group and assigning them a task does not guarantee that the students will 
engage in effective collaborative learning behavior. While some peer groups seem to 
interact naturally, others struggle to maintain a balance of participation, leadership, 
understanding, and encouragement. Traditional lecture-oriented classrooms do not teach 
students the social skills they need to interact effectively in a team, and few students 
involved in team projects or exposed to integrated working environments learn these 
skills well. 


Goal: Promote Improved 
Group Dynamics for 
Learning 

■ Teach workers/lea rners in small groups to: 

- Settle disagreements constructively 

- Resolve uncertainty in selves and others 


MITRE 
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PREDICTING EFFECTIVENESS 


We followed our initial study with the development of a collaborative learning 
environment for OMT. Our goal was to provide a web-based environment that would 
permit us to analyze peer-to-peer dialogue and tool actions in an attempt to identify the 
strength and weaknesses of a group’s interaction. The collaborative environment 
employed a sentence opener-based chat interface [16] and a shared OMT workspace tool. 
The sentence-opener interface allowed the conversational act underlying each student 
utterance to be logged; the shared OMT workspace tool permitted student tool actions to 
be recorded. We ran an experiment in which groups of three subjects used our 
collaborative environment to solve a software design problem with OMT. The results of 
our experiment [19, 21] demonstrate the potential of conversational acts to identify, for 
example, the distinction between a balanced, supportive group and an unbalanced, 
unsupportive group. Participant roles in dialogue provide another indication of group 
effectiveness. We are trying to identify problem-solving roles portrayed by the group 
members during the session to see if they might indicate progress or lack thereof towards 
successfully completing the exercise. This undertaking entails examining the relationship 
between conversational acts and problem-solving roles. 


Predicting Effectiveness 

■ Comprehend what is taking 
place in the collaborative 
_ learning environment to drive 

simulated peer 



■ Capture essence of student-to 
student interaction 
- Communicative acts and 


participant roles as indicators 


Person 

Dialogue 

Learning Skill 

Subskill 

Role 

W1 

No, a 

corporations 
made up of 
companies. 

Acknowledge 

Reject 

T-eval 

M3 

One on me 
there. A 
company has 
departments 

Inform 

Assert 

T-infog 

W1 

We have to 
draw a picture, 
riqht? 

Task 

Coordinate 

group 

process 

T-coord 
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PREDICTING TROUBLE 


The current focus of our work is on automatically detecting when trouble appears 
during group problem solving. The actions of participants, with respect to chat and 
workspace tools, provide an initial indicator of discussion and progress towards 
completing an assigned task or dealing with an obstacle. The roles of participants can 
provide a secondary indicator. 


Predicting Trouble 

■ What does it mean to say that the learning 

process is in trouble? 

- Are the participants having difficulty 
solving the problem? How can we tell? 

- Is someone talking too much and taking 
control? 

- Is someone talking too little and therefore 
not engaging in the learning process? 

- Is someone never initiating actions, only 
following suggestions? 


MITRE 
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HYPOTHESIS: PARTICIPANT ROLES CAN HELP US RECOGNIZE 

PROBLEMS 


Our goal in this research was to conduct an experiment to reveal the ways 
participants in a collaborative learning task interact and the factors that govern those 
interactions. We wanted to see if (1) the instructional roles played by members of the 
group could be deduced from machine-inferable factors about the collaboration and (2) 
whether the presence or absence of particular instructional roles indicated the 
effectiveness of the learning. Our hypothesis is that the presence or absence of particular 
roles is a powerful indicator of the status of the on-going learning process. The work we 
describe here outlines the data we are collecting on the collaboration and the role 
identification model we are constructing. 


Hypothesis: 

Participant roles can help 
us recognize problems 

■ Too many people taking on the INFO-SEEK 
role: confusion 

■ Someone playing the INFO-SEEK role and no 
one playing the INFO-GIVE role: someone is 
being ignored or lack of knowledge 

■ One person taking on the role of INITIATOR 
all the time: talking too much or taking over 
the dialog 

h One person only saying OK or YES: talking 
too little, not taking part 

■ One person switching from the INFO-SEEK 
role to being quiet or only saying OK or YES: 
frustration 

MITRE 
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THE ROLES 


We chose a set of roles developed by Benne and Sheats [1], and modified them 
slightly to suit the type of dialogues our system elicits, i.e., dialogues involving the use of 
a whiteboard and a problem-solving task. 


The roles 


■ INITIATOR (“Let's do X.”) 

■ SUGGESTER (“Let's do it this way.”) 

■ EVALUATOR (agree, disagree) 

■ OK/YES (simple acceptance) 

■ INFO-SEEKER/INFO-GIVER (questions, 
answers) 

■ CONSULTER (“Is this ok?”) 

■ SUPPORTER (“That was great!”) 

■ COORDINATOR (“Bob, why don’t you do 
this.”) 

■ SUMMARIZER (“Here’s what we’ve done so 
far.”) 


■ DRAWER 


MITRE 
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STEP 1: RECOGNIZING ROLES 


Dialogues were collected at the University of Pittsburgh and the MITRE 
Corporation to provide a corpus for study. We examined the data from five groups of 
University of Pittsburgh [20] and MITRE subjects [9]. Each group had three subjects. 
The subjects were paid undergraduates and MITRE employees. We coded the data with 
participant roles. This analysis led us to tracking the roles of participants during a 
problem solving session. The next portion of our talk describes our approach towards 
detecting underlying participant roles. In particular, we present here a discussion of 
classification and regression tree (CART) models for recognizing roles and what we have 
learned so far. 


Step 1: Recognizing roles 

■ We have chosen a set of roles 

■ We have gathered a set of dialogues and 
annotated them by hand with these roles 

■ We have build CART models to predict which 
role is being played at any given point 

- Crucial for automating the role prediction 
process 

- Vital to determining trouble spots 
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CART MODELS 


CART models are binary decision trees that can build statistical models from 
feature values. They take a series of input feature values and use them to predict another 
feature value. We use a set of features that describe the utterances in our dialogues to 
predict the role that the participants are playing. We chose a series of input feature values 
that were either captured by the system or derived from some shallow processing on the 
utterances. The features we find most useful for predicting roles at this stage store 
information concerning the general category of each utterance (a high-level 
conversational act), the specific subcategory into which the utterance falls (a more 
specific conversational act) and the person who made the utterance. For example, the 
general utterance category includes Request, which includes subcategories such as 
Request-Attention and Request-Action, and the category Motivate, which includes 
subcategories such as Encourage and Agree. 


CART models 

■ CART models: binary decision trees 

- Given a set of feature- value pairs they 
predict one value, in this case the role 

■ Inputs to the CART model (feature- value 
pairs) 

- Communicative act type (automatically 
annotated by sentence-opener buttons) 

■ General communicative act category: 
Motivate, Request, Inform 

■ Communicative act subcategory: Offer 
Alternative, Clarification, Elaborate 


Who the speaker is, who the previous few 
speakers were 

Shallow text processing MITRE 
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THE CART MODEL - THE TOP OF THE TREE 


This diagram shows the top of the CART model. Within the top portion of the 
tree, the CART model shows how it decides between the most easily recognizable 
categories, such as when the person is drawing rather than typing, and when the person is 
thanking someone or simply saying “Ok.” The “move cat” is the general category of the 
utterance, “move subcat” is the more specific subcategory and the roles are in all capitals. 
Each nonterminal node of the tree represents a question. If the answer is “yes” then the 
questioning continues down the left branch of the tree, if “no” then down the right 
branch, and either a role is chosen for that utterance or another question is asked until a 
role is finally assigned. 


The CART model - The top 
of the tree 


Move-subcat=none 

DRAW Move-subcat -Accept 


Person=Salil 

v 


n 


P.mo ve-cat-Acknow ledge 
OK EVALUATOR 


etc. 


Person-Dan 

y \ n 



P.movc-cat-Discuss 



P.move-cat=Rcquest 


INFO-GIVER 
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THE CART MODEL - DEEPER INTO THE TREE 


This diagram shows the CART model deeper into the tree. The tree shows how 
the main category of the utterance and its subcategory can indicate the role being played. 
Here we see particular roles, such as ELABORATOR or EVALUATOR, being 
determined. 


The CART model - Deeper 
into the tree 


Movc-cat-Discuss 

y/S Movc^subcat=Requcst -Confirmation 
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RESULTS 


We are using 4-fold cross-validation with the Festival Speech Synthesis System's 
CART-model building program, wagon (http://www.cstr.ed.ac.uk/projects/festival/), to 
determine the parameters for the CART model which best predict the participants' roles. 
At the moment, we do not have a large enough corpus of dialogues to decide which 
CART model we will ultimately use in our system. While we continue to gather 
dialogues, we are using these models to give us an initial indication of whether we can 
predict roles. The indicators look good: Our ability to predict roles on unseen data is 
81.5%. 


Results 

■ 4-fold cross-validation study 

■ Produced 64 CART models per choice of 
feature set 


■ Eliminated nonsensical models 

- e.g., Bill is current speaker and Mary spoke 
4 utterances ago => INITIATOR {??) 

■ Chose the best model from the rest 

- avoid overfitting 

- pay attention to results on unseen data 

■ The best tree for this data gives us 80.3% 
accuracy in role prediction 

- The features it considers important are the 
current and previous move category, 
current move subcategory, and person 
speaking 

- Underfitting: 76.1% on training. MITRE 
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CLASSIFYING INTERACTION 


Now that we have defined potential input features for an intelligent agent to 
consider when following the students’ dialogue, we need to consider triggers for 
intervention. This leads us to the problem of recognizing intervention points where the 
peer should interrupt and interact with the students. Our next step is to build a classifier 
that predicts group interaction problems in the dialogue, which means annotating the 
dialogues for different types of problems and using the role and conversational act 
information to predict the occurrence of a problem. The classifier must have information 
concerning sequences of roles played by an individual and series of consecutive roles 
played by all the participants. An initial classifier has been developed using a rule-based 
reasoning engine and trainable analysis tools. 


Classifying Interaction 

■ Apply results from small group dynamics 

■ Roles exhibited by participants 

■ Balanced participation 

■ Derive classifier from experiment data 


MITRE 
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STEP 2: DETERMINING INTERVENTION WITH CONVERSATIONAL 

ACTS/ROLES 


We followed the experimental approach Soller used in her dissertation research[20]. 
However, instead of coding knowledge sharing segments, we coded segments of 
good/poor collaboration. Our dialogue segments were often much shorter than Soller’ s 
segments (we allowed exchanges as few as five utterances). This led to poorer results. 
However, we had to work within these tight constraints since a simulated peer cannot 
help a group if it does not respond soon after a problematic dialogue sequence - detecting 
and reacting to trouble long after it occurred would only lead to more trouble! 


Step 2: Determining Intervention 
with Conversational Acts/Roles 

■ Hidden Markov Models (HMMs) classifier 

- Input: Participant conversational acts & 
roles (dialogue segment) 


- Training data: Good/Poor collaboration, 
Effective/Ineffective learning 


- Output: Probability that a dialogue segment 


category 
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DETERMINING INTERVENTION (CONTINUED) 


We developed two HMMs - one that determined whether a particular sequence of 
communicative/speech acts likely represented “Good Collaboration” and another one that 
determined whether the sequence was most likely an example of “Poor Collaboration.” 
The HMMs were trained from a portion of the data we collected and coded. Our test set 
included 1 1 Good Collaboration and 3 Poor Collaboration segments. One Good segment 
was incorrectly classified as a Poor segment and two Poor segments were incorrectly 
classified as Good segments. These results were not encouraging. 


Determining Intervention 

(continued) 


HMM Results for Good & Poor 
Collaboration (Speech Acts) 
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(Shorter bar 
indicates 
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DETERMINING INTERVENTION (CONTINUED) 


We developed two more HMMs - one that determined whether a particular sequence 
of participant roles likely represented “Good Collaboration” and another one that 
determined whether the sequence was most likely an example of “Poor Collaboration.” 
The HMMs were trained from a portion of the data we collected and coded. Our test set 
included 1 1 Good Collaboration and 3 Poor Collaboration segments. Two Poor segments 
were incorrectly classified as Good segments. These results were again not encouraging. 


Determining Intervention 

(continued) 


HMM Results for Good & Poor 
Collaboration (Roles) 
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DETERMINING INTERVENTION (CONCLUDED) 


Our HMM classifier results were inconclusive. The number of Good collaboration 
examples dominated the dialogue segments; too few Poor collaboration examples were 
available to develop a reasonable HMM. Without gathering many more dialogues, or 
revising our experiment to “promote” more examples of Poor collaboration, we are 
forced to drop HMMs as a means for distinguishing Good from Poor collaboration. 
Instead, we determined three actions to address our problem. (1) The recommendations 
from the HMMs are to be only one of many inputs to a rule-based reasoning engine that 
determines when our intelligent peer agent should intervene. (2) Other machine learning 
techniques such as neural nets should be considered. We are working currently with 
neural nets. (3) Instead of trying to classify a dialogue segment as simply ‘Good’ or 
‘Poor,’ it makes sense to look for very specific examples of collaboration. For example, if 
a student has given an utterance that needs to be replied to or acknowledged, but the 
utterance is ignored, then that indicates potential poor collaboration. We have developed 
a neural net that has been able to detect this situation with a good level of accuracy. 


Determining Intervention 
(concluded) 

■ Hidden Markov Models (HMMs) results 

- Inconclusive 

■ ‘Good’ examples dominate 

■ Too few ‘Poor collaboration’ training 
examples 

- Recommendations 

■ Use HMMs as just one input to rule- 
based reasoning engine 

■ Try other machine learning techniques 
o Neural nets 

oCART models 

■ Divide Good/Poor collaboration further 

MITRE 
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STEP 3: PREDICT INTERVENTION WITH CONVERSATION ELEMENTS 


Other conversation elements can contribute to the prediction of whether or not 
intervention is required. We looked at group agreement and disagreement, potential 
indicators of confusion, discussion topics, and comparison of the students’ evolving 
solution to the expert’s solution. 


Step 3: Predict Intervention 
with Conversation Elements 

■ Detect group disagreement 

- Communicative acts 

■ Identify confused individuals 

- Lack of participation 

- Slow contributions 

- Conversational acts 

■ Discover topics & their boundaries 

- Keyword analysis 

■ Verify solution quality 

- Model tracing MITRE 
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STEP 4: INTERVENE WITH PIERCE - AN INTELLIGENT PEER AGENT 


We developed an intelligent peer agent, Pierce, and added it to our collaborative 
learning environment. Pierce can observe up to three students working collaboratively 
over a network on a software design problem. Pierce can look for potential problems and 
attempt to intervene. 




Step 4: Intervene with 
Pierce - An Intelligent Peer 
Agent 

Human Human 

Learner Learner 


Pierce 

the peer agent 


MITRE 


Human 
Learner 


Peer Agent 
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EPSILON STUDENT INTERFACE 


We are using Soller’s modified version [20] of our collaborative learning 
interface that she originally developed for us at MITRE [19, 21]. We collected dialogues 
using Soller’s experimental procedures so we could each share the data. Our focus was 
not on knowledge sharing, however, but instead about looking for patterns of good or 
poor collaboration. The system provides a sentence-opener chat interface, an OMT 
workspace, and a new agenda tool. The agenda tool lists high-level goals to be achieved. 
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AGENT DESIGN REQUIREMENTS 


Our agent had to satisfy a set of design requirements based on the Epsilon 
architecture as well as the mode of operation desired for real-time interaction with human 
students. We also wanted the agent’s design to be flexible since we wanted the agent to 
be usable in other collaborative environments. 


Agent Design Requirements 

■ The agent must interface with NCSA's Habanero and 
the mItRE/LRDC Epsilon collaboration tool, but 
remain flexible for future integration with other 
collaborative environments. 


■ The agent must operate in real time and receive 
continuous updates from the dialogue, reaching 
conclusions and sending feedback to the participants 
on the fly. 

■ The collaborative analysis should remain domain 
independent where possible, with space to easily 
include domain specific information when helpful. 

■ The architecture must be able to interface with and 

make use of many types of analysis tools, allowing a 
combination of Al techniques to be brought to bear 
on the problem. MITRE 
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PEER AGENT ARCHITECTURE 


An initial classifier has been developed using a rule-based reasoning engine and 
trainable analysis tools. A reasoning manager controls the flow of information to the 
trainable analysis tools and the reasoning engine. Student actions, both text chat and 
drawing commands, are observed. Underlying raw data such as the communicative act, 
duration of an utterance, time between utterances, and draw object are captured by the 
agent. The agent translates the event information into a computationally useful format, 
called a “fact.” The translated facts are passed to the reasoning manager and the 
analytical tools. The analytical tools crunch the data and provide the reasoning manager 
with additional facts. The reasoning manager sends the facts to the relevant reasoning 
engines. Each reasoning engine fires rules in an attempt to reach a conclusion. 
Conclusions are passed to the agent, which if it desires, reacts by executing text chat or 
drawing commands. 


Peer Agent Architecture 
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OUTLINE OF ACTION 


When the reasoning engine fires a combination of rules that indicate an 
intervention is useful, it informs the agent. The agent sends text chat or drawing 
commands into Epsilon, as if it were a human peer, generating an intervention event. The 
dialogue box illustrates an example intervention. 


Outline of Action 

■ If the reasoning engine fires a combination of rules that indicate 
an intervention is useful, it informs the agent 

■ The agent sends commands into Epsilon, generating an 
intervention event 


well. 

51 . Milhouse : If we all have same, then your PERSON has name, age, 
employer 1 ID 

52. Pierce : Excuse me, Milhouse, but I think you might be confused. You 
should ask for help on this topic. 
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AGENT’S STUDENT MODEL DISPLAY: AGREEMENT, CERTAINTY & 

PROGRESS 

Some of the information captured by the system about individual and group 
progress can be fed back to the students through a set of gauges. The gauges provide a 
snapshot of the agent’s current estimation of the group’s and each student’s performance. 
They measure Certainty, Agreement, Percentage of Task Completed, and Dialogue 
Speed. A graph of these variables over time is also available. 


Agent’s Student Model 
Display: Agreement, 
Certainty & Progress 




Snapshot of Agent’s current estimation 
of the group's performance 


Graph of group performance variables 

as they change over time MITRE 
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FUTURE DIRECTIONS FOR PIERCE 


We are currently refining our intervention reasoner to more accurately determine 
points of intervention. We are looking at other trainable analysis techniques such as 
neural nets to process the raw event data. We are considering other features of the 
collaborative interaction as indicators. Once we have constructed a method for 
automatically predicting a group interaction problem in the dialogue and the type of 
problem, we will address as our next issue the type of response most useful for the 
electronic peer to take. Group interaction roles often seen in well functioning groups but 
absent in a particular situation are candidate roles for the peer to assume as it responds to 
the group’s needs. 


Future Directions for Pierce 

■ Parse the students’ statements looking for 
keywords and phrases to: 

- Identify current discussion topic 

- Identify changes in topic 

■ Use students’ time delay to gauge certainty 

■ Make agent’s suggestions more targeted to 
the current problem 

■ Make agent able to assume different roles in 
the dialogue (e.g. Facilitator) 

■ Use Query Manager to query a database of 
previous cases 

■ Add other methods of looking for patterns in 
the dialog 

- Neural network 

— CART model MITRE 
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CONCLUDING REMARKS 


Our previous and Soller and Lesgold’s current research investigates the 
communication among human participants jointly working on a problem in a 
collaborative learning environment. The research provides a handle on determining the 
effectiveness of collaborative discourse on a learner. The communication between 
collaborators is key to revealing the depth and level of understanding of the instructional 
domain by the group as a whole and the individual group members. Since natural 
language understanding is an on-going area of research, our work avoids full 
understanding of the discourse and instead employs a sentence-opener interface to 
provide a limited representation of each utterance by its underlying conversational act. 
Our earlier research, thus, only addresses modeling the collaborative discourse. It says 
nothing about when or how a peer or tutor should intervene to assist collaborators. Our 
current research takes this work to the next level - helping determine what is needed to 
make the collaborative interaction more effective for learning. 


Concluding Remarks 


■ Getting a handle on peer communication is 
critical to supporting learning and problem 
solving in a collaborative environment. 

■ Peer-to-peer interaction promotes learning. 
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RESEARCH AREAS 


This presentation provides an overview of the PALS (Pedagogical Agent 
Learning Systems) research agenda regarding pedagogical agent features: image, 
animation, and role. Each of these three areas is investigated using controlled 
experimental studies with large sample sizes. Our recent work in each of these areas, as 
well as our immediate future plans, will be discussed. 



Research areas 


Image : How should the agent be graphically 
V represented? 

| • Gender? Ethnicity? Realistic or cartoon-like? 

• Match learner in terms of characteristics? 

Animation : What types of animation (expressive, task- 
related) are most desirable? 

• Which are most appropriate task-related gestures? 

• What types of emotional expression is effective? 



• Role : What is the most effective role of the agent to 
_ the learner? 

• As expert? As mentor? As motivator? As learning 
companion? 

> • How can multiple age nts impact learning to promote multiple 

perspectives in complex learning situations? 


Figure 1 
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AGENT IMAGE: GENDER, REALISM, ETHNICITY 


First, in the area of pedagogical agent image, we are currently conducting a large 
scale study to investigate the effects of agent image on learning and motivation. The 3- 
factor between-subjects design (2x2x2) looks at the impact of agent image in terms of 
gender, ethnicity, and realism. Each image was created from the same basic template 
figure as a way to control for basic features, and each was validated to not be 
significantly different from the others in terms of learner perception (e.g., attractiveness, 
likableness, etc.) In this study, learners were presented with one of the 8 agents within the 
learning environment to determine effect on learning and motivation. We are also 
interested to see post-hoc whether the learners’ gender and ethnicity are related to which 
agent character s/he learns best from. A second study was also recently conducted where 
learners were presented with all of the images and asked to choose which one they would 
most like to learn from. Results from both of these studies will be available soon. 



Figure 2 
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CONTRAST COMPARISONS 


In terms of how the agent is perceived as person-like, we conducted a study with 
75 participants to isolate the impact of image presence/absence and animation 
presence/absence [11], Results of the study indicated that the presence of animation with 
the pedagogical agent was desirable for all related persona measures (as listed below) and 
particularly for “instructor-like,” which is of the most interest in a learning environment. 



Figure 3 
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VOICE AND ANIMATION 


Continuing with our preliminary work regarding agent animation, we also 
recently looked at the impact of animation presence/absence together with the effect of 
human versus computer-generated voice. We found that the presence of agent animation 
significantly facilitated improved learning from the agent, yet also negatively impacted 
learner motivation. The presence of a human voice, as predicted, led to the agent being 
perceived overall as more human-like (in terms of credibility, engaging, and instructor- 
like) but did not influence learning or motivation [13]. 
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MORE ON ANIMATING AGENTS... 


Our future work investigating agent animation will consider issues regarding the 
nature of the animation in terms of affect (e.g., emotional expression) and task-related 
aspects (e.g., gesture). Also, issues regarding cognitive load will be addressed in terms of 
how much the animations facilitate rather than distract from the learning task. 



More on animating agents.. 


• What types of animation (expressive, task-related) 
are most desirable? 

• Should the agent demonstrate both positive and negative 
affect? 

- Agent emotional expression as contributes to learning 
(Kort & Picard) 

• Which are most appropriate task-related gestures? 

- Role of agent's "body language" as instructor 

• Should agent display other types of animations? (i.e., are 
they reinforcing for learning) 

• Does presenting the agent animation simultaneously 
with voice/text reinforce or distract from the learning 
task? (e.g., cognitive load issues; Mayer) 



Figure 5 
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AGENT ROLES... 


Our third area of focus is in the area of pedagogical agent roles, and we have 
looked at the roles of expert, motivator and mentor. Through validation studies, we first 
determined an effective operationalization of each of these roles in terms of image, 
animation, affect, script, and voice [9], The Expert was designed to be expert-like in 
terms of presentation and content knowledge, whereas the Motivator lacked content 
knowledge, but supplied encouragement and ideas during the learning the process. The 
Mentor is essentially a combination of the Expert+Motivator in that he provided both 
motivation and expertise. 



Agent roles.. 



• What is most effective pedagogical role for agent? 

Mentor 



Motivator Expert 



Figure 6 
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MULTIPLE AGENTS / AGENT ROLES 


One of our most interesting results was found from controlled study of the Mentor 
agent alone versus the Expert + Motivator agents together (provided simultaneously) 
[10]. In both conditions, identical information and motivation was provided to the 
students since the Mentor was figuratively “split” into two separate agents as personified 
by the Expert and Motivator. We were surprised to find that students learned significantly 
more when the two agents were presented simultaneously (Expert and Motivator) than 
when the Mentor was just presented alone. We are terming this the “split persona effect.” 



Multiple agents / Agent roles 
(N=S 0) 1 2 



Figure 7 
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AGENTS AS REPRESENTING A ROLE OR PERSPECTIVE 


Future work regarding agent role will investigate issues such as agents as peers or 
learning companions, and as mentors/role models for under-represented groups. We have 
collected a large amount of agent-learner interaction data and will be evaluating this 
qualitatively. The final area of immediate interest is the impact of multiple agents as a 
way to promote reflection and metacognition for the learner (e.g., see [5]). 



Agents as representing a role 
or perspective.. 


• What features best define agents as mentors? 
(e.g., preparing future workforce) 

• Other roles: as peers or learning companions.. 

• Evaluating agent-learner interaction patterns 

- with agent(s) representing different roles 

• Agents as vehicles to present multiple 
perspectives (e.g., to promote metacognition) 

- Focus on ill-structured knowledge domains 

• Complex learning; cognitive flexibility theory 

• How can multiple agents best model social 
interactions to promote reflection? 


Figure 8 
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INTELLIGENT TUTORING SYSTEMS 


Over the last twenty years intelligent tutoring systems have proven to be very 
successful. From a theoretical academic perspective these systems have brought together 
disciplines like artificial intelligence, computer science, cognitive science, psychology 
and linguistics. But these systems have also been proven to be successful in a range of 
learning environments, including geometry, algebra, physics and electronics. In fact, the 
generations of systems have shown to provide learning gains, up to 1 standard deviation 
compared to students in a classroom (Corbett et al., 1999). 

The latest generation of intelligent tutoring system uses conversational interfaces. 
These systems interact with student using natural language. Students can interact with the 
conversational system using written or spoken dialog and the system responds with 
language or speech. There are various reasons it is worth investigating these intelligent 
conversational tutoring systems. Of course, we are generally good at having 
conversations and this would benefit students. But more importantly, there are 
educational advantages. Interaction with a human tutor provides learning gains up to 2.3 
standard deviations compared to classroom teachers (Corbett et al., 1999). A system that 
mimics the behavior of human tutors might therefore be desirable. 




Intelligent Tutoring Systems 


□ Intelligent tutoring systems 
computer-assisted instruction 


use intelligent 


□ Produce significant learning gains beyond class 
room environments 

□ New generation of intelligent tutoring systems 
that hold mixed-initiative conversational dialogs 
with learner 

□Atlas, Andes, and many others 



Figure 1 


253 


AUTOTUTOR 


One such intelligent conversational tutoring system is AutoTutor. AutoTutor is a 
conversational agent that assists students in actively constructing knowledge by holding a 
conversation in natural language. AutoTutor has currently been developed for the 
domains of computer literacy and physics with the possibilities of development for a 
range of other domains. The system is web-based allowing to be used on any standard 
PC. Because AutoTutor is a client based application, the user only has to download a 
small number of files to be able to run the system, thus allowing for optimal use in 
educational environments (Graesser et al., 1999; 2001). 

The system, its components and its effectiveness have been discussed in over 70 
book chapters, journal articles and conference proceedings chapters (see 
http://www.autotutor.org). This presentation will summarize some of the main 
components of the system and focus on some of the natural language processing aspects 
of the system. 



Suppose a runner is running in a straight line at constant speed, 
and the runner throws a pumpkin straight up Where will the 
pumpkin land? Explain why. 


finish each response, you will need to press the enter key 
submit button using the mouse. My responses to you will 
spoken. If for any reason, you don't understand somethir 
you can ask me to repeat it or you can look in the dialog 
The dialog history box appears on the left-hand side ofthi 
directly beneath me. Once I feel that we've spent enough 
problem, we will move on to the next problem. If you have 
Tutor: uestions, please ask the experimenter. Are you res 
begin. Suppose a runner is running in a straight line at con 
and the runner throws a pumpkin straight up. Where will th 
land? Explain why. 


Log of previous turns 


□ AutoTutor assists in actively 
constructing knowledge by holding a 
conversation in natural language 


Server Port: [4646 

|7 Show Agent |7 Show Log 


Server IP Address: |1 41 .225.1 0.1 84 


r Use Agent Bias Agent Widt 



□Client server application (web based) 

□Computer Literacy 
□Conceptual Physics 


Figure 2 
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MOTIVATIONS BEHIND AUTOTUTOR 


AutoTutor has been developed for computer assistant education. It helps students 
in their understanding of problems, concepts and terms in a particular domain. At the 
same time, AutoTutor helps teachers in providing their students with the tutoring of basic 
conceptual knowledge of a domain, leaving the real exciting challenges of tutoring for the 
human professional. 

In addition, AutoTutor allows for testing of hypotheses. Understanding of how 
students behave in certain learning environments, how they react to certain dialog moves, 
whether they prefer a particular kind of tutor, all can easily be tested using intelligent 
tutoring systems. Although those experiments might miss the human aspect of tutoring, 
they allow for a maximum of controlling various variables. 

Similarly, the development of AutoTutor has given rise to a range of research 
questions that might otherwise not have been raised. For instance, how important is world 
knowledge in tutoring? How easy is it to eliminate misconceptions? What makes 
language natural? When do language users consider dialog mixed-initiative? 


r Why AutoTutor? 

□ Computer assisted education 

□Testing research hypotheses using a computer 
model is easier and more reliable than using a 
human tutor 



□ Research questions are generated by observing 
students’ conversations with AutoTutor 



Figure 3 
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THE INTERFACE OF AUTOTUTOR 


The interface of AutoTutor can be divided into four quadrants. The first quadrant 
consists of a talking head. AutoTutor uses synthesized speech to talk to the student, with 
facial expressions and rudimentary gestures to support this speech. 

In the second quadrant a problem is presented to the student. In addition to 
presenting the problem or question, this quadrant can be used for visual information, 
including pictures, movie clips and interactive simulation. 

The third quadrant is reserved for the dialog history with previous tutor and 
student turns are presented. This allows the student to look back on previous answers and 
tutor information. Although the dialog obviously allows for retrieving this information 
too (e.g. could you repeat that?), this history gives an overall log of the tutoring dialog. 

The fourth quadrant is primarily used by the student. Although future versions of 
the system might use speech recognition, currently the student communicates with the 
system using written language. In the text box in the fourth quadrant the student can type 
answers to questions and problems presented by the tutor, various comments (e.g. I don’t 
understand) and even various kinds questions (e.g. what is the difference between 
velocity and speed?). 


Log of | 



Suppose a runner is running in a straight line at constant speed, 
and the runner throws a pumpkin straight up. Where will the 
pumpkin land? Explain why. 


Talking head 

Gestures 

Synthesized speech 


finish each response, you will need to press the enter key, or dick the ■*■ I 
submit button using the mouse. My responses to you will always be 
spoke A If for any reason, you don't understand something that I say. 

ik me to repeat it or you can look in the dialog histoiy box. 
\istoiy box appears on the left-hand side of the screen. 

Once I feel that we've spent enough time on a 
on to the next problem. If you have any q 
use ask the experimenter. Are you ready? Let's 
inner is running in a straight line at constant speed. 

1 a pumpkin straight up. Where will the pumpkin 




Dialog history with 
■ tutor turns 
■student turns 



■Presentation of the 

question/problem 


G=] 


Agent: |marco.acs 
Agent Height Bias: [o' 


Student input (answers, 
comments, questions) 
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COMPONENTS IN AUTOTUTOR 


At least four components can be distinguished in AutoTutor. First, the ‘teacher’ in 
AutoTutor organizes the pedagogical strategies (tutorial macrostructure). The ‘thinker’ in 
AutoTutor understands the meaning of the communication in general and the student’s 
answers in particular (world knowledge). The ‘dialog participant’ in AutoTutor chooses 
the right dialog moves at the right time to keep the conversation organized (dialog 
advancer network). Related to this, the ‘conversationalist’ in AutoTutor uses a range of 
strategies to provide mixed initiative dialog. Each of these components is discussed next. 



Figure 5 
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CURRICULUM SCRIPTS 


Like most human tutors AutoTutor uses a macrostructural organization of the 
tutorial. For its didactic skills it uses curriculum scripts, loosely ordered sets of skills, 
concepts, example problems, and question-answer units. These scripts allow the tutor to 
keep track of the topic coverage and follow up on any problems the student might have. 
Although the tutor does not follow these curriculum scripts step-by-step, it does keep 
track of them as a guideline for the tutorial. The student thus has much flexibility in the 
conversation with the tutor, but the tutor is able to step in when the conversation seems to 
deviate too much from the structure of the tutorial (Graesser et al. 1999; 2001). 



□ loosely ordered sets of skills, concepts, example 
problems, and question-answer units 


Curriculum Scripts 



□ keep track of the topic coverage and follow up on 
any problems the student might have 




Figure 6 
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WORLD KNOWLEDGE 


Understanding natural language requires world knowledge. AutoTutor uses Latent 
Semantic Analysis for its world knowledge. LSA uses singular value decomposition to 
reduce a co-occurrence matrix of words (or documents) to a cosine between two vectors. 

The method of statistically representing knowledge has proven to be useful in a 
range of studies (see Franceschetti et al., 2001; Graesser et al., 2002). AutoTutor uses 
LSA as follows. The system presents a question related to a problem to the student and 
the student responds to this question. AutoTutor then analyzes the accuracy of this 
answer by comparing the student’s answer with a series of expected ideal answers. This 
way the student covers a range of expectations in the tutoring sessions. The tutor allows 
the student to move to the next problem once all expectations are covered. 


World knowledge 


□ Latent Semantic Analysis (singular value 
decomposition to reduce a co-occurrence matrix 
of words (or documents) to a cosine between 
two vectors) 

□ LSA gives meaning to a student answer and 
matches that answer to ideal good and bad 
answers 


A 




Figure 7 
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MIXED INITIATIVE DIALOG 


Mixed initiative dialog is the final component we earlier identified. Mixed- 
initiative interaction refers to the flexibility of strategies applied in the interaction 
between participants by those participants. It allows for the direction and control of the 
interaction to be shifted between participants. Mixed-initiative dialog requires that the 
participants engaged in the joint project (i.e. computer and human) act in coordination 
with each other (Allen, 1999). 

Achieving mixed initiative dialog in computational environments is difficult. 
However, the question to be addressed in mixed-initiative interaction in intelligent 
systems is perhaps not whether models of human-computer interaction are mixed- 
initiative, but how the impression can be created that users are involved in mixed- 
initiative dialog. We have argued that the impression of mixed-initiative can be created 
by making the computer user believe that they are participating in a natural conversation 
(Louwerse et al., 2002). The more natural this conversation seems to be, the more likely 
the impression of mixed-initiative dialog is formed. 


^^^^Mixed Initiative Dialog 

>A flexible interaction strategy, where 
each agent (human or computer) 
contributes what it is best suited at the 
most appropriate time 

(Hearst, 1999; Allen, 1999) 



Figure 8 


260 


MIXED INITIATIVE DIALOG 


At least four features of AutoTutor contribute to its conversational smoothness 
and its mixed initiative dialog. First, dialog moves direct the conversation. Second, 
discourse markers form the glue of the separate conversational units. Third, speech acts 
identify the speaker’s intentions. Fourth, question answering gives the speaker the 
flexibility to ask questions and retrieve appropriate answers. Fifth, interactive simulation 
allows for manipulating variables in the real world (e.g. gravity) in order to understand 
various phenomena. Each of these features is discussed next. 



Mixed initiative dialog 



□ Dialog management 


□ Discourse markers 



□ Speech act classification 

□ Question answering 

□ Interactive simulation 


Figure 9 
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DIALOG MANAGEMENT 


AutoTutor’s dialog management is designed to create a collaborative learning 
environment by determining the right tutoring strategy, to facilitate the student's active 
construction of knowledge and to facilitate conversational smoothness. Several of 
AutoTutor’s dialog moves consist of a lengthy substantive contribution that prompts the 
learner for more information, that adds information, or that corrects a student error. These 
dialog moves don’t particularly make AutoTutor a conversational and mixed-initiative 
system. More specific dialog moves that for instance prompt, hint and correct and thereby 
mimic natural tutoring sessions however are highly conversational (Person et al., 2001). 



Dialog management 



□ Designed to create a collaborative learning 
environment by determining the right tutoring 
strategy 


□ Facilitate the student's active construction of 
knowledge 


□ Facilitate conversational smoothness by using 
synthesized speech, facial expressions and 
some rudimentary gestures 



Figure 10 
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DIALOG MOVES 


Seven kinds of dialog moves are used in AutoTutor’s conversation: the system 
prompts for specific information; it pumps for more information; it hints , asserts, 
corrects, repeats and summarizes. These dialog moves are organized by discourswe 
patterns in terms of their progressive specificity. For instance, hints are less specific than 
Prompts, and Prompts are less specific than Elaborations. AutoTutor therefore cycles 
through a Hint-Prompt-Elaboration pattern until the student articulates the correct answer 
(or rather reaches threshold t for covering all aspects of the ideal answer). The other 
dialog moves (e.g., short feedbacks and summaries) are controlled by fuzzy production 
rules (Graesser et ah, 1999; Person et al., 2001). Examples from the version of the system 
developed for the domain of computer literacy of the different dialog moves are given 
below in Figure 11. 


Dialog moves 

□ Pump for more information: 

“What else?” 

□ Prompt for specific information: 

“The computer output is sent to the monitor and the 

□ Hint: 

“How can you get a copy of a document? ” 

□ Assert: 


“The document can be sent to the printer.” 

□ Correct: 

a [Immediately insert the correct content after a student error] 

□ Repeat: 

“So, once again, how can you get a copy of a document?” 

□ Summarize: 

ft^So to recap, [succinct summary].” 




Figure 1 1 
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DISCOURSE MARKERS 


Discourse markers are cues that facilitate the transition between turns in a dialog. 
They support the conversational smoothness. From various studies (see Louwerse & 
Mitchell, 2003) we know that discourse markers play an important role in informal 
conversational spoken dialog. They cue participants on previous dialog moves (e.g. okay, 
right), and mark relations with upcoming units of the dialog {also, so). Given that 
discourse markers play such a an important role, the main problem lies in choosing the 
right discourse marker at the right time in order to give the impression of conversational 
smoothness. 


Discourse Markers 

• How to choose the appropriate 
discourse marker at the appropriate 
time in the appropriate place? 




Discourse markers make the dialog 
more conversational (informal 


spoken dialog) 



Figure 12 
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NETWORK OF DISCOURSE MARKERS 


To select the appropriate discourse markers at the appropriate time and place, a 
taxonomy of these markers was generated from natural dialog. By sampling a large 
number of discourse markers from a natural corpus of dialogs and substituting each 
marker for all other markers, a network of semantic relations between these markers was 
generated (see Louwerse & Mitchell, 2003). The taxonomy that could be constructed out 
of these markers was implemented in AutoTutor. Random selections from large bags of 
discourse markers were made in order to give positive, neutral and negative feedback to 
the student, and to provide the student with a conversational continuity that can be found 
in natural dialog. The selections of markers are based on the dialog management in 
combinations with scores from latent semantic analysis (LSA). For instance, a high LSA 
score can lead to positive feedback (e.g. Excellent! Well done!), to neutral feedback 
( Hmm , okay) or to negative feedback (e.g. No, not really). Using the taxonomies of 
markers allowed for the impression of conversational smoothness in AutoTutor. By going 
beyond simple yes-no feedback the impression was created that the system was not fully 
in control of the conversation, but allowed for a conversation with the student. 



Figure 13 
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SPEECH ACT CLASSIFICATION 


In natural conversation not only the meaning the meaning of a speaker’s utterance 
is important, but also the intentions the speaker has expressing that utterance. AutoTutor 
classifies utterances into a range of speech acts. Although analyzing the pragmatics of 
dialog is difficult in terms of computational linguistics, we have developed a speech act 
classifier that identifies a large number of categories using surface linguistic features. In 
addition to assertions, the classifier identifies 20 illocutionary categories, ranging from 
directives (e.g. hold on!), metacommunicative expressions (e.g. I didn’t quite hear that) 
and metacognitive expressions (/ don’t know) to 16 questions categories. These question 
categories consist for instance of definitional questions, verification questions, causal 
antecedent questions and example questions. With the classifier successfully been 
implemented in AutoTutor, the system now has ways to more appropriately respond to a 
student’s input (Louwerse et al, 2002). 

The speech act classifier is used to answer student’s questions, is used to monitor 
a student’s performance (e.g. does the student have many metacognitive problems?) and 
can be used to restructure the student’s utterance to its intended meaning. 



Speech act classifier 



□ Metacognitive statements 

□ Metacommunicative statements 

□ 16 types of questions 

□ Directives 

□ Assertions 



□ Restructure to intended meaning 

□ Monitor performance 

□ Answer questions 


Figure 14 
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RESTRUCTURING STUDENT’S MEANING 


An example of the use of the speech act classifier is the restructuring of the 
original student’s expression into an expression that better represents what the student 
tries to say. Take for instance the expression I don’t know what gravity is. Strictly 
speaking this expression is an assertion, or more specifically a metacognitive statement. 
However, what the student really wants to know is the meaning of gravity. What looks 
like an assertion is in fact a question. AutoTutor’s speech act classifier evaluates 
expressions and transforms them in an expression that better represents what the student 
intended 



Classifying Speech Acts 

Classification 





Metacommunicative 


// ...\ 


Figure 15 
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ANSWERING STUDENT QUESTIONS 


As mentioned earlier, the speech act classifier categorizes a question into one of 
16 categories, as identified by Graesser & Person (1994). This taxonomy distinguishes 
itself from other proposals in at least two respects. First, the question categories are 
defined according to the content of the information sought rather than on question signal 
words (who, what, why, how, etc.). That is, a speaker asking How can you define gravity? 
requests a different answer than a speaker asking How does gravity affect the earth’s 
orbit around the sun? despite the fact that the signal word (how) is the same for both 
questions. Secondly, the taxonomy makes a distinction between the type of questions 
asked from a learner’s perspective. One the one hand it identifies shallow comprehension 
questions that do not require a deep insight into a topic (questions 1-8 in Figure 16), on 
the other deep comprehension questions that require inferences in addition to dictionary 
or encyclopedic knowledge (questions 9-16 in Figure 16). Not surprisingly, deep 
comprehension questions and their answers will likely lead to higher learning gains. By 
identifying these 16 categories and the two types of questions in particular, AutoTutor 
can monitor a student’s educational challenges. 



Verification 


Types of questions 


(Graesser & Person, 1994) 


Is the answer 5? 

Is gender or female the variable? 

Who ran this experiment? 

What are the properties of a bar graph? 

How many degrees of freedom are on this variable? 
What is a t test? 

What is an example of a factorial design? 

What is the difference between a t test and an F test? 


Example 



Disjunctive 
Concept Completion 
Feature Specification 
Quantification 
Definition 
Example 
Comparison 
Interpretation 
Causal Antecedent 


What is happening in this graph? 

How did this experiment fail? 

What happens when this level decreases? 

Why did you put decision latency on the y-axis? 
How do you present the stimulus on each trial? 
What device allows you to measure stress? 

Why isn't there an interaction? 

What do you think of this operational definition? 



Causal Consequence 
Goal Orientation 




Figure 16 
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QUESTION ANSWERING SYSTEM: QUEST 


Identifying student questions without being able to answer these questions would 
be awkward. Using the speech act classifier AutoTutor is able to answer student 
questions by providing the information they seek. This question answering system 
(QUEST) thus allows for student initiative. Without knowledge engineering, the system 
uses question classification and latent semantic analysis to retrieve a paragraph from a 
text book that best fits the answer the student is looking for (Louwerse et al., 2002). 


V QUEST 

U Responds to student initiative by giving 
them the information they seek (question 
category can anticipate answer!) 



U Retrieves answers from a large body of text 

- Student not limited by lesson plan 

- First steps in natural language generation 

- No knowledge engineering 


Figure 17 
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STRENGTHS OF QUEST 


QUEST offers some important advantages. First, of course, it allows for mixed 
initiative dialog. The student does not only listen and answer, but can challenge the 
system with questions as well. Because QUESTS uses answers that are based on the 
information in a textbook, the student gets the answers in needs. Another important 
advantage is that QUEST performs best on those questions that are considered deep 
comprehension questions in the Graesser & Person, 1994) classification. For learning 
environments this is an important advantage, given that this is the type of questions good 
students are expected to ask. Finally, the speech act classification and latent semantic 
analyses mechanisms allow the answer to be easily embedded in the conversational 
framework. With a minimum of efforts the knowledge domain of QUEST can be changed 
and computational linguistic modules are used that are already present in the 
conversational framework of AutoTutor. 


Strengths of QUEST ^ 

U Gives detailed answers based on what the 
student learns from the text book 

^Questions considered difficult to automatically 
answer (e.g. causal antecedent, comparison 
questions) have highest performance scores 


□ The answers can easily be embedded in a 
conversational framework 


Figure 18 
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INTERACTIVE SIMULATION 


The final aspect of mixed initiative dialog goes beyond natural language 
processing only. AutoTutor uses interactive simulation environments can change 
parameters and control the simulation. All simulations were designed to be web 
deliverable and to simulate specific conceptual physics problems. However, instead of 
interactive simulations only, content specific dialogs accompany the simulations in order 
to motivate the learner to explore the simulation environment, followed by a discussion 
with the avatar pertaining to the concepts. During the simulation the system guides the 
student, while the student can take control of the simulation and dialog at any point (see 
Haynes et al., under review). 



Figure 19 


271 



EXAMPLE OF AUTOTUTOR DIALOG 


We have discussed the various aspects of mixed initiative dialog in AutoTutor. To 
give an impression of a typical dialog, Figure 20 and 21 give examples taken from 
AutoTutor log files. Figure 20 shows the tutor first presenting a problem. Instead of the 
student answering the tutor’s question, she asks another question, which the system 
answers introducing the answer and asking whether the answer is sufficient. 


Mixed initiative 
dialog sample 

Tutor: The sun exerts a gravitational force on the earth as the 
earth moves in its orbit around the sun. Does the earth pull 
equally on the sun? Explain why. 

Student: How can gravity pull on both the sun and the earth at 
the same time? 




Tutor: Very interesting point. The following will probably help. 
The sun also contributes to ocean tides, although it is less 
than half as effective as the moon in raising tides - even 
though its pull on the earth is 180 times greater than the pull 
of the moon. Why doesn't the sun cause tides 180 times 
greater than lunar tides? The answer has to do with a key 
word: difference. Because of the great distance of the sun, 
the difference in its gravitational pull on opposite sides of 
the earth is very small. The percentage difference in the 
sun's pulls across the earth is only about .017 percent, 
compared to 6.7 percent across the earth by the moon. It is 
only because the pull of the sun is 180 times stronger than 
the moon's that the sun tides are almost half as high (180 X 
0.017 percent = 3 percent, nearly half of 6.7 percent). Do A 
you understand the answer? 


Figure 20 
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EXAMPLE OF AUTOTUTOR DIALOG (CONTINUED) 


The student responds negatively to the question whether the answer is sufficient 
and the tutor follows up asking what was unclear in its answer. When the student points 
out the confusion, the system then follows up with another answer to the question. 
However, when the system has answered and starts with the next tutoring question and 
the student asks yet another question, the teacher in AutoTutor steps in. Instead of 
allowing for total flexibility in the conversation, AutoTutor now takes back the control of 
the dialog. 





Mixed initiative 
dialog sample 

Student: no 

Tutor: What is it that you don't understand? 

Student: I don't know what gravity is. 

Tutor: gravity is the attraction between objects due to mass. 

Tutor: Okay, see if you can get this. Newton's Third Law refers to 
the forces exerted by one body on another ? 

Student: What is Newton's third law 

Tutor: Wait a minute, answer my question and then we'll do yours. 
Let me repeat. Newton's Third Law refers to the forces exerted 
by one body on another ? 

Student: body 


Figure 21 
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OVERALL STRENGTHS OF AUTOTUTOR 


AutoTutor has various strengths (see Graesser, 1999; 2001; Louwerse et al. 2002; 
Person et al., 1999). It is an existing system that requires a minimum of computer 
specifications. Furthermore, AutoTutor is not domain specific. Because of the various 
natural language processing modules discussed in this presentation, AutoTutor requires 
only a minimum of adjustments for a new domain (approximately three months for a 
basic version of the system). More specifically, only the knowledge domain and the 
curriculum scripts need to be changed for the new domain. Authoring tools have been 
developed to facilitate this process. Furthermore, the system adapts to student. Studies 
have shown that AutoTutor uses different pedagogical strategies and different frequencies 
of dialog moves for high ability versus low ability students, thus allowing for a certain 
extent of user modeling. Finally, AutoTutor is conversational, so much that experiments 
have shown that subjects cannot distinguish between dialogs controlled by AutoTutor and 
dialogs controlled by human tutors. However, the system does not just control the dialog. 
It takes control whenever this is necessary, but allows the student to contribute what is 
best suited at the most appropriate time, providing mixed initiative dialog. 



Strengths 




□ Existing, compatible system 

□ Learning gains 

□ Domain flexible 

□ Interdisciplinary (cognitive science, psychology, 
linguistics, education, physics, etc.) 

□ Language generation (curriculum scripts) 

□ Real dialog (bystander Turing test) 

□ Adaptive to student ability 

□ Mixed initiative dialog 


Figure 22 
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COMING TO TERMS WITH SOCIAL CONSTRUCTIVISM 


Throughout this workshop, social constructivism - the idea that people construct 
what they learn through social interaction with other people (Palincsar, 1998; Wilson, 
1996) - has been lurking in the background of our discussions. 

I want to talk today about how I have begun to come to terms with the ideas of 
social constructivism in my own work. It has been a process, one that is still underway. 
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OUTLINE OF TALK 


I started out life as a cognitive psychologist, and was taught to regard “learning” 
as a matter of information processing mechanisms. However, I worked for the first 18 
years of my career in industry (at IBM’s Watson Center) and for the next 10 as a 
university professor (at Virginia Tech). In both cases, I was repeatedly impressed that 
learning is more than a matter of sequences of learning events and practice. 

Learning occurs when people carry out authentic tasks, in realistic contexts - 
including committing, diagnosing, and recovering from errors, when they are able to 
explore and discover on their own initiative, reflect upon and draw lessons from their 
activity, when they are able to work with others, socially model the practices of others, 
and share meaning with others. I have detailed my own itinerary as a researcher on 
learning in a series of books - Carroll, 1990, 1998, 2000. 
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My itinerary 

♦ Learning requires authentic activity 

■ meaningful goals & realistic context, errors 

♦ Learners “make meaning” 

■ problem-solving, discovery, reflection 

♦ Learning is social 

■ enculturation, communities of practice, multiple 
perspectives 


Figure 2 
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RESEARCH THEMES 


Social constructivist ideas about learning have direct implications for designing 
teaching practices and technology. First, designers should collaborate directly and deeply 
with domain experts. This means more than just knowledge engineering for the given 
domain. It means understanding teaching and learning in the domain. 

Second, designers need to provide learners with open-ended tasks. People cannot 
be expected to make meaning out a dreary, over-programmed practice. Significant 
learning activities always incorporate discovery. 

Third, designers must incorporate collaborative learning into new teaching 
practices and technologies. Whether “agents” can be effective surrogates in collaboration, 
and whether purely asynchronous interaction with others is sufficiently stimulating are 
current technical questions. 
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\4 tan “““ 


Resultant research themes 

♦ Collaborate with domain experts 

■ to identify authentic tasks 

♦ Emphasize open-ended learning activities 

■ to enable meaning making 

♦ Support and investigate collaborative learning 

■ because learning is social 


Figure 3 
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LEARNING IN A NETWORKED COMMUNITY (LiNC) 


In 1995, my group began a long-term collaboration with public school divisions 
in southwestern Virginia to explore the feasibility of employing Internet collaboration in 
high school and middle school science, as part of NSF’s “Networking Infrastructure for 
Education” initiative. A specific problem faced in these rural areas is that often there is 
insufficient critical mass for science courses, which of course adds to the inequity already 
faced by students from these areas. The technologies and approach we developed are of 
course more general than this, and in fact are widely applied in so-called distance 
education. 

We worked with 6 teachers for more than 5 years. Our program eventually 
involved more than 400 students. Our initial technical objective was to integrate 
synchronous and asynchronous collaboration. It was obvious even then that this would be 
a major challenge for getting distance learning visions to work. But along the way, we 
encountered a large set of other issues, some of which are far more challenging. 
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Learning in a Networked Community 

♦ Participatory design involving 6 middle and 
high school science teachers for 5+ years 

♦ Project-based collaborative learning, four 
school sites, and 400 students 

♦ Integration of synchronous and asynchronous 

♦ Also curriculum design, activity awareness, 
teacher collaboration 
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THE VIRTUAL SCHOOL 


This is the first version of our Virtual School system. It clearly makes the point of 
how important awareness and coordination are in collaborative work. In center of the 
screen, is the shared notebook; the sharing is at the level of sections. Collaborators can 
embed annotations in the shared document. The green lock indicates that we are working 
on the Robot Pictures section; the red lock indicates that others are working on the 
Introduction 

A video window helps keep us aware of remote collaborators. Chat and Email 
(not shown here) are also used to coordinate. A Notice Board tool (in the lower left) 
displays recent project chronology, an inventory of shared objects (in this case, just the 
robotics notebook), and a roster of group members - indicating who is 
available/unavailable. (See Isenhour et al., 2000.) 


r^r \\ Inko 



group 

events Chat and email not shown. 
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EARLY OUTCOMES 


Our in the objective in the LiNC project was to explore a new paradigm for 
collaborative, project-based science. This involves both collaboration outcomes and 
learning outcomes. On both fronts, we had encouraging but inconclusive results. We 
were able to over-come many remote collaboration problems. We were able to analyze 
the challenge of activity awareness and a variety of technical approaches to managing it 
(Carroll, Neale, et al., 2003). 

With respect to learning outcomes, we relied primarily on our partners in the 
public school systems. They concluded that this intervention was successful in promoting 
the educational goals by which they are measured. Two of the 6 teachers that worked 
with us continued using our software in their classroom even after our NSF funding 
expired. We have continued to work with these school systems in a series of further 
projects. 
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TEACHER COLLABORATION 


One of our most important outcomes in the LiNC project was something we were 
not looking for. To make collaborative learning work, teachers have to work together too. 
The challenges of technology with respect to the culture of teaching and the school is 
well-known. But analogous issues exist with respect to collaboration (Tyack & Cuban, 
1995). 


In the LiNC project, teachers collaboration, and responsibility-taking more 
generally, increased as we went along. We were able to identify developmental patterns 
and trajectories in the roles that the teachers we worked with played in our joint project. 
A key mechanism, we believe, is serious and long-term commitment to participatory 
design, an approach in which “users” and “developers” of technology share equal power 
and responsibility. (See Carroll et al., 2000.) 
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Long-term Participatory Design 


♦ 


♦ 


Four “stages” 

■ Practitioner-Informant 

■ Analyst 

■ Designer 

■ Coach 

Developmental 

■ Relatively stable knowledge, skills, attitudes 

■ Transitions resolve conflict in activity 

■ Increasing responsibility, scope of action 
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CURRENT TECHNICAL DIRECTIONS 


Our work is continuing along four directions. First, we have tried to enhance the 
software infrastructures that enable systems like the Virtual School. To support seamless 
collaboration and learning, teachers and students need to be able to configure such 
systems. 

Second, we are continuing to investigate activity awareness; we see this as a 
major issue in enabling collaborative work systems. 

Third, we have shifted some of our focus from students to teachers. Supporting 
continuous teacher professional development is both a prerequisite for the sort of 
educational vision we are pursuing, and a possibility afforded by this vision. 

Fourth, we are developing and investigating new collaborative learning activities; 
as has always been true, new technologies both enable and require new approaches in 
human activity. 
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Current active threads 

♦ Enhanced collaborative SW (inc EUP) 

♦ Understanding activity awareness 

♦ Facilitating teacher professional development 

♦ New collaborative learning activities 

8 
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SOFTWARE TOOLS 


Today, the World-Wide Web is the default infra-structure for educational 
collaborations. However, the Web is a publishing infrastructure. With support from ONR, 
we developed the Content Object Replication Kit (CORK) to enable lightweight 
collaborative interaction. 

In the object replication approach, changes in a local object are broadcast to 
subscribed collaborators and to a central server in order to update all replicas. On the one 
hand, this approach requires less network bandwidth than broadcasting the objects 
themselves, and, on the other hand, it enables asynchronous collaboration, not possible 
with screen sharing approaches. 
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Software tools 


♦ CORK: Content Object Replication Kit 

■ Component architecture for collaborative tools 

■ Detect/propagate changes via lightweight messaging 

■ Integrate synchronous and asynchronous interaction 

■ Demosatjava.cs.vt.edu 
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CORK EXAMPLE 


Here is an example. The client at the left updates the map, and a change object is 
broadcast both to the collaborating client, at the right, and to the central server, depicted 
at the bottom right. Activity in the collaborating client is very minimally affected at all; 
the clients are synchronized on a ‘when possible’ basis. We described this in more detail 
in Isenhour et al. (2001). 
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BRIDGE 


More recently, we have developed a tool layer to provide file system and user 
interface services for CORK objects. This layer, called BRIDGE - Basic Resources for 
Integrated Distributed Group Environments - is intended to make it easier for people to 
collaborate directly via the Web. 
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Software tools 


♦ CORK: Content Object Replication Kit 


♦ BRIDGE: Basic Resources for Integrated Distributed 
Group Environments 


■ Mapping Web & interactive UIs onto CORK objects 


■ Naming & stmcturing collections of CORK objects 

■ Composing objects for Web publication, graphical 
markup, & interactive environments 


■ General services for replicated objects: versioning, 
awareness, notification 

11 
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EXAMPLES OF BRIDGE 


These are examples. Bridge provides a file system view of CORK objects. It 
provides awareness information, such as the currently-subscribed users of an object. And 
it provides standard ways of rendering CORK objects, such as maps and tables. 



Figure 12 
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ACTIVITY AWARENESS 


Activity awareness is our sense of the status, needs, and directions of an on-going 
collaborative project. It is more than a matter of who is looking at what data now. 
Physical proximity provides a great deal of transparent support for activity awareness. 
We believe that collaborative must do this too. We are exploring four directions: First, 
awareness information should be created incidentally; it should not become a further task 
for users. In our work, it is created as a side effect of ordinary document transactions. 
Second, awareness support should be integrated with work. In our designs, it is embedded 
in workspace views that encountered in carrying out primary tasks. Third, awareness 
information should be public, viewable at a glance. We are experimenting with larger- 
screen displays for this purpose. Fourth, the tasks required to maintain awareness should 
be delegated. We are exploring the use of agents as a mechanism for this. 
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Activity Awareness 

♦ Incidental creation of awareness information 

■ Via document transactions (emails exchanged, 
versions saved) 

♦ Integrated presentation of awareness 
information 

■ Workspace views that embed activity awareness 

♦ Large-screen public access to all group activity 

♦ Awareness agents 
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VIRTUAL SCHOOOL 2 


Our continuing development of the Virtual School, supported by NSF’s 
“Information Technology Research” initiative, provides some examples of these 
directions. Some of the tools are familiar from the earlier Figure 5: Availability of 
collaborators, shared objects, a group chat tool. In this design more attention has been 
given to supporting emergent work organization. The to-do list in the lower left allows 
project members to keep track of goals. 

More significantly, all project documents are organized in a time line view, which 
emphasizes the availability of previous versions, but also places the current versions in 
time with respect to deadlines (Carroll, Neale, et ah, 2003). 



Figure 14 
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VIRTUAL SCHOOL: CONCEPT MAP 


This is another view, with the timeline rolled up to make room for a document 
window, in this case a concept map representation of the content work of part of the 
project. Concept maps are very widely used in high school and university education. In 
future work, we plan to make the concept map a first-class representation of the project 
workspace, that is, an isomorphs of the timeline itself. 
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VIRTUAL SCHOOL: LARGE-SCREEN DISPLAY 


The Virtual School 2 was fielded this fall in two classrooms. In February, we also 
fielded the large-screen, public interface that integrates across groups. This view consists 
almost entirely of timelines. An activity ticker at the top of the display cycles through the 
groups, providing a recent highlight of each group’s recent activity. We are using it to 
explore the utility and other consequences of providing an integrative overview of all 
project activity in the connected classrooms. 

A paper describing the Virtual School 2 system was submitted to the ACM UIST 
Conference (Ganoe et al., 2003 submitted). 
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AWARENESS AGENTS 


The fourth approach to supporting activity awareness that we are investigating 
involves limited agents that can carry out awareness-supporting missions such as “tell me 
when Captain Martin returns to the Wardroom”. Such agents allow people to off-load 
some of the attention that would have been directed at maintaining awareness, but they 
also may enhance awareness through reminding and task engagement (see Farooq et al., 
2002). As part of an ONR project, we have experimented with these techniques in the 
context of a Web-based MOO (multi-user domain) called MOOsburg (Carroll et al., 
2001 ). 
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TEACHER PROFESSIONAL DEVELOPMENT 


One’s first impulse in educational research is to try to improve student 
performance. But the longer-term challenges and the bigger payoffs related more directly 
to teachers. Students stay in school 16-18 years; teachers stay 30-40 Years and impact 
hundreds of students. 

Many problems in teacher professionalism derive from the fact that innovation in 
technology and practice is not regarded as standard operating procedure, or in some 
cases, rewarded at all. As mentioned earlier, teacher collaboration is not seen as a regular 
function of teachers. Teachers in service know these things must change, and many of 
them are looking for ways to change themselves. 

We conceive of the current context in schools as similar in many respects to that 
in other organizations facing analogous transformations. With NSF support, we are 
investigating the use of knowledge management techniques and technology to support 
new approaches to teacher professional development. 



Coming to terms with 
Social Constructivism 


Teacher Professional Development 

♦ The culture of schools 

■ Lack of reward for innovative practices 

■ Quasi-professional job status 

■ One’s classroom as haven of control 

■ Diminished community status 

♦ Teachers recognize that this must change 

♦ Knowledge management approach 

18 
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LEVELS OF KNOWLEDGE SHARING 


A simple way to look at this is that teachers need help in moving to “higher” 
levels of knowledge sharing. Today, they may share tangible resources. For example, in 
one school we worked with an enormous pendulum was shared; after all, who would 
even want a 30 foot high pendulum. It is more difficult to share plans, for example. Each 
teacher has a philosophy, has plans and resources, and may not want to expose these to 
others or to enter into a social process of negotiating adaptations. 

One goal of teacher knowledge management is to move sharing to the level of 
prototypes-in-use, a term we use to describe pedagogical activities populated with data 
and experience. Prototypes-in-use are more personal, more evocative, richer, and more 
inviting to collaborators. (See Carroll, Choo, et al., 2003.) 
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Levels of knowledge sharing 


♦ Tangible resources 

■ Very large pendulum apparatus 

♦ Activity plans 

■ Schedule, reading list, & sample questions for a 
regional academic competition club 

♦ Prototypes-in-use 

■ Stream monitoring tools & activities, data tables, data 

gathering protocol, descriptions of the environment, 
photos, discussion forum, and project notes 19 
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EXAMPLE: DIFFUSION OF GOOD IDEAS 


One approach we are using to try to facilitate higher levels of knowledge sharing 
is to help teachers develop their existing good ideas in a medium that makes the ideas 
more sharable. A teacher had run a reading/discussion activity for several years using 
command-line MOO software that presented a house with rooms that students could visit 
as they discussed books. Eventually, she was unable to run this ancient software. 

We helped her to present the same basic activity using our CORK and BRIDGE 
software. This was easy to achieve and resulted in a far more refined user interface 
approach, as in figure 20. Moreover, it was extensible and reusable. As shown in the 
figure, this prototype-in-use can be recruited, modified, and re-purposed by other teachers 
for other similar activities. 
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EXAMPLE: BRING STAKEHOLDERS TOGETHER 


Another approach we have used is to support the development of new activities 
that bring a variety of local stakeholders together. The pandapas.org site is a good 
example. Pandapas Pond lies along the shared border of the two southwestern Virginia 
school divisions e are working with. Teachers from those school divisions, along with US 
Forest Service personnel, and others interested in the national forest can come together 
through this site, sharing activities and data. 
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Pandapas: bring stakeholders together 
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NEW COLLABORATIVE ACTIVITIES 


Whenever technology does not merely change the ways that we do things, it also 
presents us with new things to do. Part of our focus has been to investigate new 
collaborative learning possibilities that are enabled by the technologies we are 
investigating. For example, we are investigating new approaches to case-based learning 
in the context of a 3rd year undergraduate course in Usability Engineering, and we are 
investigating cross-generational informal learning through the collaborative construction 
and analysis of visual simulations of community dilemmas. 
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m ' 

Collaborative activities 

♦ Analyzing usability case studies 

■ 3rd year CS/UE class 

■ What is a good case? What is a good case-based 
learning activity? 

♦ Building visual simulations of community 
dilemmas 

■ informal learning 
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USABILITY CASE STUDIES 


We have worked with industrial partners to create hypertext descriptions of 
several software development case studies. Students can browse the design documents 
and products, for example, the Web page designs for garden.com in Figure 23. They can 
study the entire design process from requirements analysis through activity design, 
information design, interaction design, documentation design and usability testing. They 
can carry out compare and contrast analyses of different case studies. This been highly 
successful in our own classes at Virginia Tech, as an adjunct to our textbook on usability 
engineering (Rosson & Carroll, 2001). We have recently begun a national test-bed project 
with NSF’s Division of Undergraduate Education to further develop and investigate this 
approach. 
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CROSS-GENERATIONAL COLLABORATIVE LEARNING 


Many aspects of our work have benefited from the technological and social 
infrastructure of the Blacksburg Electronic Village, a well-established, Web-based 
community network. Like many people, we noticed that children often grow up far from 
their own grandparents, but still in proximity to community elders. Senior citizens in 
Blacksburg are a well-organized and active group, and agreed to work with us on a 
project in which middle school children and seniors learn visual simulation programming 
techniques through designing simulations that depict community dilemmas (fighting in 
the school yard, noise in residential areas, bullying, and so forth). The projects are a 
forum for enhancing everyone’s technology literacy and skills, and also for discussing 
different points of view regarding community issues. (See Rosson et ah, 2002). 
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CACULATING COMMUNITY WELLNESS 


One direction we are going with this project is to aggregate a sense of community 
wellness from the ensemble of community simulations pertaining to a variety of places 
throughout the community. We are imagining having people discuss not only isolated 
community simulations, but systems of contemporaneous simulations. For example, 
solving a conflict between residents regarding the noise created by a block party might 
balance the tension generated by a standoff among cliques at the middle school. 
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MOOSBURG 


This vision exploits another piece of our collaborative infrastructure: MOOsburg, 
a Web-based MOO (multi-user domain) modeled on the layout of the actual town of 
Blacksburg, and navigated through use of a zoomable interactive map widget (lower 
right). Critical mass is a key limiting factor in creating rich information and activity 
spaces. Part of our vision is to have endeavors like the Virtual School, pandapas.com, the 
usability case studies, and community simulations play off one another in the shared 
community space of MOOsburg. (See Carroll et ah, 2001). 
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FOLLOW-ON WORK 


In the immediate term, we are pursuing several further lines of technology 
development and empirical investigation. First, we are developing security and federation 
approaches, which are required for external implementations based on CORK and 
BRIDGE. Second, we are developing and assessing knowledge rubrics beyond the 
timeline that we explored in our recent Virtual School work (see Carroll, 2003, Rosson, 
et al.). Thirds, we are integrating the case study approach to project-based learning with 
our teacher professional development work, and focusing more attention on teachers as 
end-user programmers. Our vision requires that teachers directly create learning objects 
using BRIDGE and further tools. Finally, we are continuing to broaden our investigation 
of informal and community-based strategies to teaching and learning about Information 
technology. 
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Plans and Start up 

♦ SW: security, ad hoc networking 

♦ KM representations beyond timeline 

■ concept maps, edit wear, social nets 

♦ pedagogical case studies for TPD 

♦ more focus on teacher EUP 

♦ informal learning, community-based IT 
training, m-education 
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TECHNOLOGY INNOVATION FLIGHT: EDUCATION AND TRAINING 
TECHNOLOGY APPLICATION PROGRAM 


This presentation is about an Air Force Education and Training Command 
(AETC) initiative called the Education and Training Technology Application Program 
(ETTAP). 


Studies and Analysis Squadron 
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Education & Training 
Technology 
Application Program 

Dr. Ken Levi 
April 2003 


Integrity - Service - Excellence 


Figure 1 
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INNOVATING FOR TOMORROW 


ETTAP is all about bringing new technology into the classroom. 


Innovating for Tomorrow 

Aerc 

• “...AETC must continue to improve the 
effectiveness and efficiency of training to ensure 
our high standards and capabilities into the 
future. ” 

» AETC Strategic Plan 2001 

• “We must leverage new, more complex 
technologies and develop new innovative ways to 
enhance mission accomplishment. ” 

» Lieutenant General John Hopper 
AETC Vice Commander 
AETC IT Strategic Plan, Volume II 


Figure 2 
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OVERVIEW 


The briefing will cover the purpose of ETTAP; the process by which Air Force 
classrooms can obtain ETTAP funds; some examples of past and current projects, 
including a live demo of what we would consider the ideal; how we define a good 
project; the role of the project officer; how we evaluate new classroom technologies, 
some of the ETTAP programs that are going on nationwide; and finally, how you can 
reach the ETTAP program for more information. 




Good Proji 


9 



Project Officer 



Figure 3 


313 



PURPOSE 


The purpose of ETTAP is getting new technology into the classroom fast. This 
involves identifying those technologies and then evaluating how well they do. 



ETTAP is designed to identify and evaluate 
innovative technologies improving the efficiency and 
effectiveness of education and training. 


Figure 4 
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TYPICAL ETTAP PROCESS 


Suppose there’s a course at one of the twelve bases in the Air Education and 
Training Command; let’s say it’s a course on Leadership. And the instructor goes to a 
trade show or reads a journal and discovers a new technology - a computer game, for 
instance - that he thinks will help his students learn better/faster/cheaper. Well, ETTAP 
gives him the seed money to buy it. So, he purchases 30 copies of this computer game 
and gives it to his students. We then spend the next six months to a year to see how it 
pans out. Does it, in fact, help students to learn Leadership better, faster, or cheaper? If 
it does, then great. If not, well - at least we tried. And we didn’t waste a lot of time and 
money in the process. 



Typical ETTAP Process 

(ETTAP - Education & Training Technology Applications Program) 



Howto An innovative use of 
train/educate COTS to 


better, cheaper, improve/enhance 

and faster? training/education 



Idea works 
great! 



Problem: Without hard numbers to justify, good ideas may 
get dropped, and bad ideas may get implemented 


Figure 5 
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MICROFLIGHT 


Here are some examples of ETTAP projects past and current to give you an idea 
of what I’m describing. 

MicroFlight is a PC based flight simulator used to familiarize new pilots with 
flight maneuvers. It helped T-37 pilot trainees at Laughlin AFB TX become proficient 
faster. 





MicroFlight 


PC-Based Flight 
Simulator 

Navy Training Shell 
Additional features 

• T-37 Characteristics 

• MS Flight Simulator 98 

• Laughlin AFB Terrain 
Helped students to Chair 
Fly Missions 
Reduced Time-to-MIF 


Figure 6 
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AIR TRAFFIC CONTROL (ATC) VIRTUAL REALITY 


The Air Traffic Control Virtual Reality system helped trainees at Vance, Altus, 
Luke, and Randolph AFBs simulate extreme hazards which they would rarely, if ever 
encounter in real life. 


Air Traffic Control Virtual Reality^ 

A£W 


• ATC upgrade training 

• Replaced aircraft models 
on sticks 

• Trained infrequent tasks 

• Decreased upgrade 
training time 

• Small system footprint 
(Deployable) 



Figure 7 
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COST EFFECTIVE VIRTUAL REALITY 


Cost Effective Virtual Reality (CEVR) is VR without the headset. What you see 
is a three dimensional object like a hologram that jumps out at you from your PC screen. 
Instructors at Sheppard AFB TX used CEVR to teach the components of the Mark 84 
bomb. 


V Cost-Effective Virtual Reality 5 


• Munitions Maintenance 
Training 

• Potential Benefits 

• Replace Physical Mock-ups 

• No VR Headset Required 

• Uses Standard Classroom 

• Tested Efficacy of 
Technology 

• Revealed critical 
limitations 

• Saved valuable dollars 



Figure 8 
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F-15E VIRTUAL REALITY 


Also at Sheppard, the F-15 Virtual Reality system does require a headset. It 
teaches students maintenance on the F-15E fighter aircraft and saves the schoolhouse the 
considerable expense of having the actual jet on hand for the students to train on. 



F-15 Virtual Reality 





• F-15 E, Safe-for- 
Maintenance Training 
Interactive Courseware 

• Determine if Virtual 
Reality could replace a 
Hardware Trainer 

• Students used Joystick 
and T rigger to carry out 
maintenance tasks 

• Decreased 3-level 
Certification Time from 24 
hours to 2 hours 



Figure 9 
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SCHEMATIC POWER BROWSER 


Schematic Power Browser is one of our most widely adopted projects. It allows 
instructors to show complex schematics on a Smart Board and manipulate those 
schematics by hand, taking them apart, moving them around, and magnifying them at 
will. 


V Schematic Power Browser 

■■■ Atnrc 

• Combines Overhead 
Projector, Video Projector, 
and Smart Board to 
Explore Complex 
Schematics 

• Potential Benefits: 

• Increased Comprehension 

• Faster Understanding 

• Increased Test Scores 

• Streamlined classroom 
management/prep 



Figure 10 
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ELECTRONIC CLASSROOM 


We built a comprehensive multimedia electronic classroom at Vandenberg AFB 
CA, complete with PCs, LAN, Internet connectivity, smart boards, overhead projection, 
online testing, and instructor control of individual student workstations. It enables Space 
Officer trainees to receive all their instruction and all their exams electronically. 



Electronic Classroom 



• Automated Paperless Fully 
Electronic Training 

• Simultaneously Combine 
Lecture, Computer Based and 
Web Based Training 

• Enable Distance and 
Distributed Academic Learning 

• Automated Testing and 
Records 

• Benefits 

• Establish Standards for AETC 
Electronic Classroom 

• Reduced instructor time to 
grade examinations 



Figure 11 
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ELECTRONIC BOOKS 


Students at Keesler AFB use electronic books to store 80,000 pages worth of 
hardcopy textbooks, manuals, and regulations in one lightweight convenient device. 
One of the best features of E-Books is update. In the past, if Keesler instructors wanted 
to make a change in a student manual, however slight, they had to wait 5 Vi weeks for the 
print shop to create new hardcopies, and pay an average of $30 per document. With E- 
books, updates can be accomplished in seconds versus weeks, and the cost is free. 



Electronic Books 


5 

actc 


• Replaced Paper Media 
with Electronic Media 

• Potential Benefits: 

• Saves Reproduction Costs 

• Faster Course Updates 

• Surveyed E- Book 
Capabilities 



Figure 12 
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N’GRAIN EXAMPLE 


Finally, here’s an example of what we would consider an ideal ETTAP project. 
It’s a commercial-off-the-shelf - or COTS - product that allows an instructor to display 
on a big screen, a three dimensional view of any hardware or piece of equipment. 

■ He can rotate it to show how it looks from any angle. 

■ He can disassemble it piece by piece, and then reassemble it. 

■ Or, he can challenge his students to reassemble it themselves. 

■ He can zoom in to show closer views. 

■ He can even cut it down into cross sections in order to show what it looks 

like inside. 

This example is a watch. But the demo also includes a rifle, an aircraft, and a 
bomb. Of course, the instructor can choose whatever piece of hardware he wants. 

The total cost ranges from about $15,000 to $50,000 if you want the contractor to 
build a whole course for you. But this is a lot cheaper than having an actual aircraft to 
demonstrate on. Or, it’s a lot safer than having an actual bomb. 



N’GRAIN Example 


5 



Figure 13 
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A GOOD ETTAP PROJECT 


So, what makes a good ETTAP project? We ask ourselves these questions: 

■ Is it new technology? Is it off the shelf? 

■ Does it make learning better, faster, or cheaper? 

■ Can we test it out in a year? 

■ Can we use it for other classes? 



Command/AF- 
wide impact 


A project that can 
be completed in 
one year 


Ideas for which 
results can be 
quantitatively reported 


Good ETTAP Project 5 


An innovative concept 
using commercial-off-the 
shelf hardware and/or 
software m 


Figure 14 
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PROJECT OFFICER TASKS 


If the answer to these questions is “yes,” then here’s what the classroom instructor 
need to do. 

1. Submit a one-pager to HQ AETC (pass out SF) 

2. If we approve it, then, we’ll ask you to go into more detail 

3. Help us test it out in one or more of your classes 

4. Every three months, tell us how things are going 

5. Then, at the end of a year, tell us how things worked out 



Project Officer Tasks 






• Prepares short form 

• Prepares long proposal 

• Assists SAS with study 
plan preparation 

• Collects data for study 

• Submits quarterly reports 

• Submits summary 
study report 

Reference: AF! 3@~22‘18 


Figure 15 
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STUDIES AND ANALYSIS TASKS 


Evaluating each project takes about a year and requires several stages of detailed 
planning. Once we think we’re ready, we start collecting data from classes of actual 
students in AETC. In the end, it all comes down to a final report, which declares whether 
or not the technology was effective. 


W Studies and Analysis Tasks 9 

AS7C 


Evaluation Planning & Execution Process 



Figure 16 
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CURRENT ETTAP PROJECTS 


This is just a big picture view of all the ETTAP projects we have going on nation- 
wide. 



Current ETTAP Projects 



CHEYENNE 


SHEPPARD 


ALTUS 


MOUNTAIN 


-Hand-Held 

Diagnostic 

Ultrasound 

-Voice 

Recognition for 

Curriculum 

Development 


-KC-135 Boom 
Operator 
3-D Visuals 


-Data Mining 
for Weather 
Education 


•Crew 

Resource 

Management 

Debriefing 

Tool 


•C-130 Crew 

Resource 

Management 


VANDENBERG 


•ICBM 

Operational 

Procedures 

Emulator 


MAXWELL 


-Electronic 

Essay 

Evaluation 

-Artificial 

Intelligence 

Tutoring 

-Automated 

Transcript 

Transfer 


PHOENIX ANG 


-Ground 
Theater Air 
Control 


•Fire Protection 

Virtual 

Environment 


KIRTLAND 


RANDOLPH 


•MC-130 EW 
Embedded 
Simulator 


-Advanced 

Cockpit 

Trainer 


/keesler 

TYNDALL 

/ ^Relational 

•F-15 Cockpit 

Knowledge 

Procedures 

Modeling 

Simulator 


Figure 17 
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ETTAP POCs 


And here are some contact points for further information. 


Ms Betty Hartsfield 
AETC/XPRT 

DSN 487-7390, FAX 487-2537 
email: betty, hartsfield@randolph.af. mil 

Capt Anthony Lujan, SAS/IT 
DSN 487-4917, FAX 487-6895 
email: anthony.luian@randolph.af.mil 



ETTAP POCs 


9 


Web addresses: 

http://www.aetc.af.mil/xp 

http://www.aetc.af.mil/sas/itaf 



Figure 18 
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IMAGINATION 


Imagination - that’s what the ETTAP program is about. 


“Imagination is more important than 
intelligence. Intelligence is about the 
past and the present, but imagination 
is about the future.” 


Replenishing the Combat Capability of America's Air Force 



9 



Figure 19 



Integrity - Service - Excellence 


Figure 20 
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AETC Technology Integration and Development 


Capts. Garald Egts and Phong Nguyen 
HQ AETC/XPRT 

Air Education and Training Command 
United States Air Force, Randolph AFB, TX 
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Air Education and Training Command 


AETC 
Technology 
Integration and 
Development 

Capt Gary Egts 
Capt Phong Nguyen 
HQ AETC/XPRT 

Integrity - Service - Excellence 



Figure 1 



Innovating for Tomorrow 



• ..AET C must continue to improve the 
effectiveness and efficiency of training to ensure 
our high standards and capabilities into the 

future.” 

» AETC Strategic Plan 2001 

• “We must leverage new, more complex 
technologies and develop new innovative ways 
to enhance mission accomplishment.” 

» Lieutenant General John Hopper 
AETC Vice Commander 
AETC IT Strategic Plan , Volume II 


Figure 2 


333 



Overview 



• Mission and Responsibilities 

• Mission Areas/Programs 

• Education and Training Technology 
Application Program (ETTAP) 

• Technology Transition 

• AF Battlelab Support/Other Initiatives 

• Generalized Operations Simulation 
Environment (GOSE) 

• Points of Contact 


Figure 3 



Mission/Responsibilities 



Improve the effectiveness and efficiency of A ETC 
training and education by identifying, evaluating, 
and advocating new technological applications 

• Explore the intersection of learning and technology 

• Plan/Advocate for technology insertion and innovative practices 

• Fund conceptual applications 

• Test ideas, concepts, and technologies 

• Provide quantitative and qualitative ROI assessments 

• Influence program managers and senior leaders 

• Connect technologies (tools) to solve user problems 

• Program Managers determine their program requirements and 
whether the interventions “fit” 

• Senior leaders make the programming trade-offs 


Figure 4 
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Mission/Responsibilities 



• AETC focal point for AF Battlelabs, Science & Technology, 
Modeling and Simulation (M&S) 

• Represent AETC at OSD, DoD and USAF level meetings 

• Keep program managers abreast of changes 

• Maintains AETC resources in AF M&S Resource Repository 

• Provide technology education to AETC units 

• Functional Utilization and T raining Workshops 

• Crossfeed technological advances/achievements 

• Gather internal educational and training requirements 

• Identify and document command’s inefficiencies 

• Seek cost-effective technological solutions 


Figure 5 



Mission Areas/Programs 



Recruiting/Accessions 

Policy/Programming 

USAF Chief Scientists 
Group 

Aircraft Maintenance and 
Munitions 

Logistics Training 
Combat Support Training 
Modeling/Simulation 

Communications and 
Information Training 

AF Battlelabs Liaison 


Education 

Technology Planning 
Air and Space Training 
Medical Training 
Flying Training 
Classroom Technology 
Education and Training 
Technology Application 
Program (ETTAP) 

Applied Technology 
Council 


Figure 6 
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Innovation Migration Process 

“End State” 




Execution + Innovation + Funding = Mission Capability 


Figure 7 
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Our Customers 


Kirtland Randolph Altus 


Fairchild 


Hill 


Vandenberg 


Luke 



Sheppard 

Goodfellow 


Figure 8 
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Overview 



• Mission and Responsibilities 

• Mission Areas/Programs 

• Education and Training Technology 
Application Program (ETTAP) 

• Technology Transition 

• AF Battlelab Support/Other Initiatives 

• Generalized Operations Simulation 
Environment (GOSE) 

• Points of Contact 


(ETTAP - Education & Training Technology Applications Program' 



• Command’s innovation process (AETC Instruction 36-221 8) 

• Identify and evaluate new technologies to solve education/training 
inefficiencies and deficiencies 

• Provide timely adoption and integration of proven technologies, 
capabilities, concepts, and processes into AETC planning and 
acquisition activities, organizations, and operations 

• Harness new technologies to provide better, faster and 
cheaper training 

• Since FY 2000: 160 projects proposed, 39 approved 


Figure 9 



ETTAP 




Figure 10 
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What Makes a Good 
ETTAP Project? 




Innovative proof of concept 
using commercial-off-the-shelf 
hardware and/or software 
High-tech ideas at low cost 
Ideas for which results can be 
quantitatively measured 
Project can be completed within 
1 year 

Project can be used 
comma nd/AF wide 


Scoring Criteria 

Measurable Redundant 

Need Innovation 

Benefits Financial Risk 

Development Risk 


Figure 11 


¥ Typical ETTAP Process 


til 




How to 
train/educate 
better, cheaper, 
and faster? 


An innovative use of 
COTS to 
improve/enhance 
training/education 



1 ^ 

Problem: Without hard numbers to justify, good ideas may 
get dropped, and bad ideas may get implemented 


Figure 12 
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ETTAP Process Flow 


HQ AETC 


Provide Technology 
Awareness Education 




Figure 13 


V ETTAP Process Flow (con’t) 


BASE-LEVEL 





HQ AETC 


SAS Writes Study Plan 


MOA Signed 


X 


Executes Study Plan 


Provides/Collects baseline data 


Implements ETTAP proposal 


Provides Quarterly Status Reports 



Project is Funded 


Interim Analysis and 
Advice 




Figure 14 
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V ETTAP Process Flow (con’t) 5 


BASE-LEVEL 


HQ AETC 


Project Officer 
Writes Study Report within 
30 days of 

Study Completion 0 ' 



SAS Analyzes Data 


SAS Writes Final Report 
within 30 days of Receipt 
of Study Report 


T 


■ 


Figure 15 



FY ETTAP Schedule 


3 

o 

r~ 

m 


O 

-< 

o 


Call 


short forms 
+ Short forms due 

+ Committee meets 

# Selection/Disapproval letters 

# Request long proposals 

# Long proposals due 

♦ Committee Validation 

♦ XP Approval 

♦ MOAs signed 
♦ Funded 


Call message, short forms + 

Short forms due# 

Committee meets + 
Selection/Disapproval letters # 


Request long proposals 


Long proposals due + 
Committee Validation ♦ 
XP Approval # 
MOAs signed 
Funded 


Figure 16 
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Advanced Virtual Schoolhouse 

(Example ETTAP Proposal) 




Students 


Live Instruction 


Learning 

Management 

System 

(Blackboard) 


Self Paced 
Instruction 


Data Collection 
& Reporting 


AFIT 

Advanced 

Virtual 

Schoolhouse 


Figure 17 


Recent Validated Proposals 5 

AETC 

• Spatial Disorientation Countermeasures 

• Replace Vista Vertigon Spatial Disorientation (SD) trainer 

• Evaluate the level of technology needed to effectively train SD countermeasures 

• Student-Centered Multimedia 
Learning 

• Evaluate “student-centered” multimedia learning environment 
in education and training areas 

• Restructure teaching methods using T ablet PC, wireless netw 
web camera, smart board, and on-line testing technologies 

• Automated Marksman Scoring 
System 

• Reduce weapons training time associated with Basic Military T raining 

• Determine effect of remote target analysis and scoring on training 
time, performance scoring time, and safety 

• Wireless Electronic Performance 
Support System 

• Provide standardized expert instruction to students in medium and high-risk activities 

• Determine effectiveness of wireless and non-wireless hand-held devices that provide 
standardized training videos to students 




Figure 18 
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Current ETTAP Projects 



for Curricufun 
Development 


PHOENIX ANG 
•Ground 
Theater Air 
Control 


• Fire Protection 
Virtual 
Environment 


LITTLE ROCK 

• C-130 Crew 
Resource 
Management 

MAXWELL 

• Electronic 
Essay 
Evaluation 

•Artificial 

Intelligence 

Tutoring 

•Automated 

Transcript 

Transfer 


CHEYENNE 


MOUNTAIN 


• Crew 
Resource 
Management 


AFIT 

• Data Mining 
for Weather 
Education 


VANDENBERG 
• I CBM 

Operational 

Procedures 

Emulator 


MC-130 EW 
Embedded 
Simulator 


GOODFELLOW 


SHEPPARD 


• Hand-Held 
Diagnostic 
Ultrasound 
•Voice 


ALTUS 


• KC-135 Boom 
Operator 
3-D Visuals 


Tool 


RANDOLPH 

•Advanced 

Cockpit 

Trainer 


KEESLER 
• Relational 
Knowledge 
Modeling 


TYNDALL 
• F-15 Cockpit 
Procedures 
Simulator 


KIRTLAND 
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Current ETTAP Projects 


PHOENIX ANG 
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Overview 



• Mission and Responsibilities 

• Mission Areas/Programs 

• Education and Training Technology 
Application Program (ETTAP) 

• Technology Transition 

• AF Battlelab Support/Other Initiatives 

• Generalized Operations Simulation 
Environment (GOSE) 

• Points of Contact 



Figure 21 



ETTAP Process Flow (con’t) 



BASE-LEVEL 



HQ AETC 


Technology Transition 
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DRAFT 


AFRL 


AETC REQUIREMENTS PROCESS 



l NO 



Figure 23 
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Virtual Private Network 



• AF Officer Accession T raining School 
(AFOATS) needed to reduce 
operating costs for connecting 143 
AFROTC detachments to Maxwell’s 
“.mil” website and network 

• Detachments required access to 
update cadet information and obtain 
course curriculum 

• VPN met or exceeded expectations 

• Reliable system (99.999% uplime rate) 

• The VPN encryption and routing process 
was “transparent to the users" and the 
VPN concentrator was “One of the most 
reliable boxes on the network" 



internal LAN VPN Server 


Air University technical personnel 


Figure 25 



Overview 



• Mission and Responsibilities 

• Mission Areas/Programs 

• Education and Training Technology 
Application Program (ETTAP) 

• Technology Transition 

• AF Battlelab Support/Other Initiatives 

• Generalized Operations Simulation 
Environment (GOSE) 

• Points of Contact 


Figure 26 
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AF Battlelab Support ® 

4fW 


• AETC provides AF Specialty Code qualification 
training to all operational AF personnel 

• As Battlelab initiatives are developed and tested, the 
AF must understand the impact that these initiatives 
will have on training and its associated costs 

• AETC assists AF Battlelabs in determining these 
training impacts 


Figure 27 
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AETC’s Process to 
Assist the Battlelabs 



• AETC established four possible involvement 
categories (an internal classification) 

Category I - Battlelab Initiative referred to AETC for lead 

Category II - Battlelab Initiative requires Training Impact 
Assessment 

Category III - Battlelab Initiative requires no AETC action 
until AF Requirements Oversight Council approval 

Category IV - Battlelab Initiative requires no AETC action 

• Process managed by AETC Battlelab Liaison 
Team (HQ AETC/XPRT and AETC SAS/IT) 

• AETC intention is not to supplant but to assist 

• AETC is the AF expert on training/education 


Figure 29 



Other Initiatives 



• Participation in bi-annual Applied Technology Councils 

• Identify potential impact on AETC missions 

• Assess potential impact: Significant, Moderate, Minimal 

• Potential to leverage technology for AETC mission need 

• Technology in Education and Training Conference 

• 28-29 May 03, Live Oaks Civic Center, San Antonio, Texas 

• Co-hosted by HQ AETC and American Society for Training and 
Development, San Antonio Chapter 

• Establishing Community of 
Practice using the Air Force 
Material Command’s 
“AF Knowledge Now” website 



Figure 30 
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Overview 



• Mission and Responsibilities 

• Mission Areas/Programs 

• Education and Training Technology 
Application Program (ETTAP) 

• Technology Transition 

• AF Battlelab Support/Other Initiatives 

• Generalized Operations Simulation 
Environment (GOSE) 

• Points of Contact 


Figure 3 1 


V GOSE Overview 

0 



• Purpose 


• Need 


• Concept 


• Advantages 


• Status 



Figure 32 
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Purpose 



• Explore emerging technologies for potentially 
solving training system concurrency and 
sustainment problems 


Figure 33 



The Need 



• Flying Training Simulators 

• Sustainment difficult ~ concurrency expensive 

• Contract Training Simulation Services (CTSS) 
unaffordable to AETC 

• Space Operation Training Simulators 

• Unsupportable 1-off engineering models 

• Onerous Multiple Instructor Requirements (MIR) 

• Maintenance Training 

• Actual hardware usually unavailable 

• Mock-up training devices very expensive 

• Concurrency is difficult 

• Some resident training reduced to academics only 


Figure 34 
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Concept 



• Scaleable and modular software-based simulation 
architecture to simulate operational environments 

• 3-D Immersive Simulation 

• Haptic/Kinesthetic 
Interfaces 

• Applications 

• Maintenance Trainers 

• Console systems 

• Space Ops 

• Air Operations Center 

• AirCrew 

• Cockpit systems 



Figure 35 


•/ Advantages 5 

AE7K 

• Common software environment to maintain 

• No system hardware requirements to create 
concurrency problems 

• Weapon system block upgrades reduced to software 
changes - shortened modification lead-times 

• Enables use of virtual or real crew members 

• Allows easy distributed team training 

• Reduce or eliminate 1 -of-a-kind simulators 

• Ability to introduce intelligent agents 

• Needed technologies are feasible in the near term 
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Status 



• Developing data and briefing trail to validate AETC 
need 

• Defining mission needs and operational requirements 

• Initiating planning 

• Will use a spiral development to provide incremental 
capabilities 

• Leverage existing research efforts over next 2-3 years 

• Begin Advanced Technology Demonstration (ATD) 
effort in FY06 with Air Force Research Laboratory as 
funds become available 


Figure 37 



GOSE Summary 



• Training System related shortfalls will cause 
serious degradation of AETC training capability in 
the near-to mid term 

• The GOSE concept can address these shortfalls 
and potentially provide affordable upgrade paths 
for future capabilities 
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Points of Contact 



HQ AETC/XPRT 

(210) 652-7840, FAX (210) 652-2537 

Project Managers, Instructional Technologies 
Capt Gary Egts, garald.egts@randolph.af.mil 
Capt Phong Nguyen, phong.nguyen@randolph.af.mil 

ETTAP Program Manager 

Ms Betty Hartsfield, betty.hartsfield@randolph.af.mil 
Web address: http://www.aetc.af.mil/xp 
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Standards-Based Approach for 
E-Learning Technologies 


Jerry West 

Advanced Distributed Learning Co-Lab 
Alexandria, VA 
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Standards-based Approach 
for e-Learning Technologies 

April 2, 2003 
Jerry West, D.Sc. 

ADL Co-Lab 
Alexandria, VA 
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“...the world’s increasing dependence on 
lifelong access to new knowledge is 
transforming the landscape of higher 
education and forcing the academy to 
rethink virtually all of its systems and 
traditions... 


Strategic Choices for the Academy: How the Demand for Lifelong 
Learning Will Re-Create Higher Education. By D. J. Rowly, H. D. 
Lujan, & M. G. Dolence, Jossey-Bass Publishers, March, 1998. 
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But How To Avoid Shovelware? 


“This form of structure. . . encourages 
teachers designing new products to simply 
“shovel” existing resources into on-line 
Web pages and discourages any 
deliberate or intentional design of learning 
strategy.” (Oliver & McLoughlin, 
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I’m as mad as 
hell; I’m not 
going to take this 
anymore! 

“Network”, 1976, 

shouted by Howard 
Beale (Peter Finch) 
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• Railroad tracks 

• Lightbulbs 

• Power outlets 

• Beer bottle caps 

• Phone lines 

• The Internet 
Standards for 
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Advanced Distributed Learning: Vision 

Provide access to the highest quality education and training, tailored 
to individual needs, delivered cost effectively, anywhere and anytime. 
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SCORM Organization 



Content Packaging (from IMS) 


BOOK 1 : 
The SCORM 
Overview 


SCORM 


BOOK 3: The 
SCORM Run Time 
Environment 


BOOK 2: The SCORM 
Content Aggregation Model 


Content Struct Lie (derived from AICC) 


Launch. Cormuiication API (TromAICC) 


(Meta data XM Binding and Best Practice (from IMS) 


m t 
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• Advance the state-of-the-art in learning technologies^^!.. 
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Worldwide Network for Learning 
Technology 



ISO/IEC JTC1 SC 3 6 
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IMS Australia 
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How the SCORM Fits 


Instructional Capability, 
Modeling & Simulation, 
Intelligent Tutoring, 
Economic Models, 
Policy, 

R&D, etc. 




Technical 

Specification 


SCORM is necessary, but not sufficient ... 

m. 


Figure 15 


Why We Need Common e- 
LearningSpecifications 

Surprising as it seems, before June 2000 we could not 

• Move a course from one Web-based Learning Management 
System (LMS server) to another. 

• Run or reuse course content across different LMS systems 
(multiple proprietary vendor tools). 

• Create searchable learning content or media repositories 
across different LMS environments. 


Figure 16 
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Traditional e-learning model 



Figure 17 
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Learning Objects 


“A learning object can be 
compared to a LEGO™ which 
can be snapped together with 
any other piece. In the world of 
learning content, we start to see 
the opportunities that would 
result if we were able to have the 
same standards and capabilities 
to reuse and assemble or 
disassemble content drawn from 
any source.” 



— Wayne Hodgins 


IEEE - Learning Technology 
Standards Committee 
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Sharable Content: Learning Objects 



Shared Object 
(Sharable Content Objects) 



Figure 19 
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Leveraging SCORM 1.2 
Content Package Structure 



Sequencing information can be 
associated with the organization and / 
or each item in the organization. 
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SCORM 1.3 Content Aggregation 




Figure 22 


365 






















ADL 

Advanced 

Distributed 

Learning 

uiwiw.odlnct.org 

SCORM Benefits 


Anyplace 

Anytime 

• Self tests 

- get it right during development 


aiJfHHWKE 

Advanced 

Distributed 

• Interoperability tests 


Learning 

ADL 

- reality test maturing products 


• Third-party certification programs 


www.adlnet. org 

1 li;;li QuulltA 
education 

individual looming 

Anylimc 

Anyplace 

Advanced 

Distributed 

Learning 

- trusted criteria for claims 


wwwadlnet.^g. 


_jgL 


Figure 23 


The Missing SCORM piece 
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Before IMS Simple 
Sequencing Specification 


Activity A 

If status == “satisfied” 

Then goto C 

Else goto B 


> Conditional branching 
logic is built into the 
Resource 

> Can’t easily reuse 
Resources, because 
of ‘hard wired’ 
dependencies 
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After IMS Simple Sequencing 



If status != “satisfied” 


If status == “satisfied” 


GotoC 


If status == “not satisfied” 
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Definition: Activity Tree 



LMS Sequencing behavior is 
described in terms of traversing the 
nodes of the tree to determine which 
activity to deliver to the learner 
Activities can be aggregated and 
organized into a conceptual tree 
Activities are nodes in the conceptual 
activity tree 

There is a default traversal path that 
can be modified through the 
association of sequencing rules by a 
learning designer 

Traversal is triggered by a sequencing 
request, which is triggered by the 
learner through navigation events, or 
by the delivery system 
Sequencing rules are evaluated at 
runtime and can be conditional based 
on tracking status 
Activities are be delivered one at a 
time 

Auxiliary resources can be associated 
with activities - _ 


Figure 28 
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is a represented as a set of teaming aaivtles with cleaned 
m to achieve the desired sequencing behaviors The 
sec defines the information that must be defined for activities r 
re the sequencing behavior desired by die Learning Designer 
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An activity tree can be Interchanged 
between systems (e g Authoring Tools and 
LMSs) using a Content Package This is 
not the only way it can be eschangad, but it 
is the only way currently specified by IMS 
The Sequencing Spec defines extensions to 
the Content Packaguig Spec to allow 
sequencing Information u be packaged with : 


of explicitly defined sequencing behaviors 
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The Sequencing Loop 
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There are many ways to usi 

The further down, the harder they become 


Hula Hoop. Listed below are somi 

of the techniques “ 

There are more techniques on the next page 

hoaiiimj - First, stand straight with your feel apart so that 
outside of each foot is aligned with the outside of each 
iulder Slightly bend your knees With both hands, position 
the Hula Hoop so that it is around your waist and against your 
back before spinning. 

Bring the hoop up with both hands so that it is around 
ick and against the back of your neck and gently start it 
nnning around your neck Again move your upper body slowly in 
ular motion, same as Basic Hooping 


Basic H 

the outs 
shoulde 


For this trick, start the hoop rotating around your 
knees. Open both arms wide to balance yourself Try to maintain 
a consistent speed and move your knees slightly back and forth 
in a circular motion. 

H«minr|i> - Start with Basic & Knee Knocker, keep up the speed 
and try to stand on one leg. Concentrate to stay balanced 
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Classroom 


Learning Management 
System 


Convergence of Knowledge 
Management and traditional e- 
learning 

Based on learning object model, 
reusability, and team 
development 



Live, Virtual 
Classrooms 


LCMS 


Learning Content 
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The Scalability and Usability Of 
Asynchronous Learning Networks 


Shawn Foley 

Pennsylvania State University 

“Technologies and growing pressure to provide quality learning 
experiences on-campus, are transforming higher education.” 
(Anderson, Garrison, 1998). 


Figure 1 



Objective s 

Determine the relevance to the 
audience 


Discuss challenges encountered 
Example: Take a look at a course 

Present the results of a quantitative 
benchmarking study 

Discuss the importance of pedagogy 
to curricular design 


Figure 2 
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Objectives 

Examine technology tools and 
services that allow the course to be 
administered 


Examine some methods in which our 
course was designed and developed 

Provide recommendations 
■ Questions/Discussion 


Figure 3 



Relevance to the Audience 


Figure 4 
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Example: Brief History and 
Unique Challenges 

■ The School of 1ST: Penn State 


■ 1ST Program Focus 

■ Educate students to be leaders and problem solvers 

■ Educate students to understand and apply technology 

■ Sensitize students to the people and policy issues 

■ Educate students using real-world problems 

■ Engage students in the realities of information technology in 
the workplace. 

■ Put the students in teams to deal with their exploration, and 
push them to build their oral and writing skills. 

■ Knowledge Acquisition, Generation, and 
Integration 


Figure 5 



Common Challenges 

■ Reaching several stakeholders and 
educational institutions 

■ Reaching a broader student audience 

■ Decreasing instructor workload 

■ Designing for various learning 
environments 

■ Acquiring and Managing Content 

■ Course Evaluation and Revision 

■ Maximizing benefits of technology 


Figure 6 


379 



Example: Pedagogy 

■ PBL Methodology and Hands on Skills 

■ Combination of theoretical approaches 

■ Cognitivist/Constructivist 

■ Behaviorist 


■ Collaborative Learning/Team-based 

■ Guiding and Generative Learning 

■ Solution: Hybrid Approach (Mixing online 
learning with important traditional residence 
based components) 


Figure 7 



Example: Course Tools and 
Learning Support Services 

■ Allow for modularity and multiple delivery 

■ Vary components depending on instructors 
preferences 

■ 1ST 220: Telecommunications and 
Networking 

■ Course Interface 

■ Course Communication Space (LMS) 

■ Communication Roadmap 

■ Solutions Learning Exchange 


Figure 8 
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Example: Multiple Assessment 
and Learning Support 

■ Course Structure: Modular Components 

■ Problems at the foundation 

■ Self and Team Evaluation 

■ Hands On Labs 

■ Discussion Activities 

■ Module Quizzes 

■ Learning Support: Factoids, URL Lists, 
Graphics, Interactive Media, External 
Resources 


Figure 9 
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Number of Sections 


Number of Sections 



Figure 1 1 



Enrollments 

Enrollments 



Figure 12 
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Study: Asynchronous Course 

■ 6 month rolling enrollment 
asynchronous course 

■ Case-Based Learning (No Teams) 

■ Multiple Assessment Strategies 

■ Multiple Communication Methods 

■ 4 Office Hours a Week 
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The Stu dy 

■ Secondary Research 

■ Time span 1 2 weeks, 32 - 45 students 

■ Learner to Instructor 

■ Instructor to Learner 


■ Tracking Time on Task for Instructor 

■ Type of activity and time spent in hours 

■ Type of communication and time spent in minutes 

■ Tracking Communication Methods 

■ Type and number of communications sent 

■ Type and number of communications received 


Figure 14 
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Time Spent on Task per Week 

■ General Communication: 1 hour 

■ Discussion Activities: 2 hours 

■ Labs: Vz hour 

■ Module Quizzes: 0 hours 

■ Problem Assignments: 3 hours 

■ Office Hours: 2.5 hours 

■ General Administration: 1 hour 

■ Total: 10 hour 


Figure 15 



Percentage of Time Spent on 
Task per Week 


Time Spent in Hours 



□ Communication 

■ Discussion Activities 

□ Labs 

□ Problem Assignment 

■ Module Quiz 

□ Journal 

m Office Hours 

□ General Administration 


Figure 16 
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Total Time Spent in Minutes 
on Communication 

■ Course Email: 286/533: 2121 min or 35 V 2 hours 

■ Personal Email:33/43: 265 min or 4 V 2 hours 

■ Telephone: 6/23: 290 min or approx. 4 % hours 

■ AOL Instant Messenger: 5/2: 65 min or approx. 1 
hour 


■ Bulletin Board (Required): 21 4/9: 2743 min or 45 % 
hours 


■ Bulletin Board (Optional): 39/15: 92 min or approx. 1 
V 2 hours 


■ Face-to-Face: 1/0: 60 min or 1 hour 


■ Totals: 584/625: 5636 min or approx. 93 hours 


Figure 17 



Percentage of Time Spent 


Time Spent in Minutes 



□ Course Email 

■ Personal Email 

□ Telephone 

□ AOL IM 

■ Bulletin Board 
(Discussion) 

□ Bulletin Board 

□ Face-to-Face 
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Comparing Models: 

Traditio nal Lecture 

■ Time on task for a traditional residence based 
course (4 + hrs.) 

■ 45-50 Students, Instructor: Lecturer, Presenter 

■ Class meets 3 times a week for one hour 

■ Discussion and lecture occurs in class 

■ Class meets 1 hour a week for the lab 

■ Problem solutions need to be graded 

■ Labs need to be graded 

■ Module quizzes are graded automatically, scores 
need to physically posted, tests are handed back 


Figure 19 



Comparing Models: 
Asynchronous Course 

■ Time on task for an asynchronous rolling 
enrollment course (3 + hrs.) 

■ 45-50 Students, Instructor: Mentor, Guide 

■ Class never meets, communication occurs 
virtually 

■ General communication, 1 hour a week 

■ Discussion activities, 2 hours a week 

■ Problem solutions need to be graded 

■ Labs need to be graded 

■ Module quizzes are graded automatically scores 
are posted electronically 


Figure 20 
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Pedagogically Sound 
Technol ogy Solutions 

“At the foundation of any effective online education or training is the 
understanding of present pedagogical practices. Whether those practices 
have been in place for years or they are brand new.” 

(Foley, 2002) 


■ Pedagogy is the foundation 

■ Instructional strategies should support 
your pedagogy 

■ Technologically sound solutions should 
support the instructional strategies and 
ultimately your pedagogy 


Figure 21 



Step 1 : In structional Design 

■ Analysis (Needs, Learner, Context, 
Task) 

■ Design - Content is King 


■ Development - Infrastructure is god 

■ Implementation - Infrastructure is god 

■ Evaluation - Is it working? Can it be 
improved? 


■ Revise Accordingly 


Figure 22 
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Step 2: Customized Process 
and Meth odology 

■ The Key is Process and Methodology 


■ Should be customized to meet the needs of 
the institution 


■ Should be based on strategies and models 
that are already proven valid and reliable 

■ Should be modular, scalable and replicable 

■ Should increase the effectiveness of present 
teaching as well as save time and money 


Figure 23 



Instructional Design: Course 
Design and Development 

■ Learner Analysis and Learner Profiles 

■ Project Plan/MOU 

■ Goals and Sub-Goals 


■ Justification Statement (Establish Need) 

■ Collect Background Data/Plans For Future 
(Flow has the course been taught in the past 
what should change?) 

■ Assumptions (target audience, delivery 
structures, possible educational context and 
delivery strategies) 

■ Key Challenges 

■ Resources, Timeline and Budget 


Figure 24 
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Instructional Design: Course 
Design and Development 

■ Content Outline (Content is King) 

■ Instructor Books, Websites, Other Resources 

■ Subject Matter Experts (Instructors) 

■ Develop Content Collection Strategy 

■ Review Content in Chunks 


■ Brainstorm Instructional Strategies (focus 
on tough concepts) 

■ Development (flowcharts, storyboards, 
rapid prototyping, Going Live!, Evaluation) 


Figure 25 



■ Content Management Systems (Databases) 

■ Content is King 

■ Infrastructure is god (manage content) 

■ MySQL/PHP, FileMaker Pro, DB2, Oracle, Access 

■ Learning Management Systems 

■ Angel, WebCT, Blackboard, First Class 

■ Focus on the learner to learner, learner to 
instructor, and learner to content communication 

■ The Web 


Figure 26 
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Recommendations 


■ Treat Content as King 

■ Based on Objectives 

■ Chunk in terms of Learning Objects 

■ Chunk in terms of Information Objects 

■ Develop content without considering the final 
outcomes or instructional strategies 


■ Plan for scalability and modularity 

■ Technology is constantly in transition plan 
accordingly 


Figure 27 



Recommendations 


■ Focus on learner 

■ Education and learning should 
not suffer 

■ Communication is key 

■ Be aware of digital divide issues 

■ Be sensitive to perceptions 

■ Learner and Instructor support is available 

■ Develop a process and a methodology 

■ Take things one step at a time 

■ Remain flexible 


■ Be organized and document everything 


Figure 28 
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OVERVIEW OF LEARNING CHALLENGES 


In framing the question, “How can advanced technologies transform learning”, we 
take a broad perspective, looking not only at how learning technologies can be improved 
but also asking how advanced technologies might change what we need to know, how we 
access knowledge, and how we work together. Given this broad perspective, we choose a 
number of technologies being developed at IBM that, we believe, have the potential to 
powerfully impact learning. 

The technologies we discuss are organized by how they might address important 
learning challenges (listed below in Figure 1). These include the challenges of 
transforming tasks to better fit our competencies; better situating knowledge so that what 
we need to know is “ready to hand”; reducing language as a barrier in cases such as 
illiteracy or when translation is required; finding and indexing relevant information for 
the solution of problems, including video and other rich media; gainfully collaborating 
with others; and reflecting and learning from our experiences. This list is not meant to be 
exhaustive (e.g., numeracy is missing), but to be broad and representative. Further, it 
highlights areas where IBM and others have made significant recent advances. 


1 . 

Transform 

Revise how a task is performed. 

2. 

Situate / Transfer 

Recall and place in context the knowledge needed for the 
problem at hand. 

3. 

Language 

Literacy. Translation. 

4 

Reflect 

Learn from experience. Formalize what you know so it can 
be shared with others. 

5 

Search / Index 

Ability to find the information you need. Create indices so 
rich media can be retrieved. 

6 

Collaborate 

Find people to work with to solve a problem. Devise 
mechanisms that enable people to work together. Revise 
social systems for improved performance. 


Figure 1 
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TRANSFORMING THE TASK: 2D FACE TRACKING 


Technologies transform how we accomplish tasks through the coordination of our 
memory, cognition, fine motor skills, vision, etc. A dramatic example of one such 
transformation is provided by a system that allows quadriplegic users to access computer 
applications (see Figure 2). A face tracking system allows them to move the screen 
cursor by moving their nose, replacing the need to manipulate a mouse with their hand 
[1]. Clicking and double-clicking are accomplished by voice commands. In this way the 
technology transforms the control of the mouse from hand to nose and voice. This 
technology transforms what actions are needed to accomplish a task. The implication is 
that educators become, in part, designers who must decide which actions should be used 
to accomplish what tasks. Of course, such decisions can have dramatic ramifications. 
Over two millennia ago Plato (writing in the voice of Socrates) worried that the replacing 
memorization with the use of written texts would diminish his students’ powers of 
memory. Indeed writing is believed to have resulted in reduced memory in literate 
societies, as it promotes a dramatic reorganization of cognition, memory, and perception 
in the accomplishment of many tasks and, in the competencies and habits of those 
facilities. Thus, through our advanced technologies we reconfigure our role in the 
accomplishment of tasks and thereby what needs to be learned and performed. 



Figure 2 
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SITUATE INFORMATION USING “THE EVERYWHERE DISPLAY” 


A primary way to help learners is to provide situated knowledge. For example, 
contrast driving assisted by a road map versus road signs. Using the map involves tracing 
a route on it and then aligning that route with one’s current view (e.g., figuring out where 
one is on the map). In contrast, road signs are embedded into our view of the world. The 
often difficult step of coordinating representational frames is skipped. The Everywhere 
Display is a powerful technology that can situtate a wide variety of information by 
overlaying it on (or augmenting) the world [2], In effect, this creates an interactive world 
of signs to guide our actions. 

The Everywhere Display accomplishes this by projecting its images onto an 
arbitrary surface and then recognizing our gestures as we interact with the images and 
virtual controls it projects. For example, in Figure 3, a picture is being dynamically 
drawn on a conventional table. Thus, the use of projection and gesture recognition 
transforms the table into an interactive device. The picture is being drawn as pointillist 
drawing using M&Ms where the computer guides the user in the placement of the 
M&Ms. At the moment shown below, the user is guiding what portion of the picture 
should be projected by a sweeping gesture. The computer’s visual analysis of this 
gesture is shown inset in the lower left. Thus, the ordinary interaction via keyboard and 
mouse has been replaced with gestures that are recognized using advanced vision 
algorithms and the ordinary screen has been replaced by projecting images onto arbitrary 
surfaces. 

In this way the physical world has been painted over with a layer of 
information — the computer has situated its information into our perspective. In a word 
the world has been infomated. This technique can have dramatic implications for 
learning and performance. Imagine asking for help in using a copy machine and that help 
being displayed directly on the machine, animating what should be pressed and 
recognizing and correcting your actions as you try to comply. This advance is very 
significant since we know a primary stumbling block to successful use of what has been 
learnt is applying it to novel situations (the so called “transfer problem”). The 
Everywhere Display shows how that transfer can be assisted as computers augment the 
world with layers of information. 
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SITUATE USING “PHOTOGRAPH BASED LOOKUP” IN INFOSCOPE 


Most categorizing systems use some sort of index to organize their contents. For 
example, books in a library are organized by the Dewey Decimal system. Indices are 
very powerful because they allow us to quickly look things up and to organize things 
with respect to one another. But they also introduce learning issues in that they pose a 
level of separation between the thing being sought and the means of asking about it. For 
example, in classifying a tree the taxonomic key might ask you to whether it has leaves or 
needles. These decisions become a problem when it is difficult to describe what you see 
in terms of the key. In contrast, the InfoScope system uses the world itself as the index 
for the information you’re seeking [3], Using a handheld computer you take a picture of 
your surroundings, say a building. It wirelessly transports it to a server along with your 
current geographic coordinates (using a global position system) and it looks it up in a 
database and returns information about that building (e.g., when it was built, what it 
looked like originally, its architecture plans). Alternatively, InfoScope can translate 
signs. For example, in Figure 4, below the signs have been translated revealing the name 
of the train station. This means of information lookup makes the world itself an index to 
our information stores, rather than using an intermediate representational language. 



Figure 3 
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LANGUAGE LEARNING ENVIRONMENTS 


The shortfall of skilled workers needed for the global knowledge economy has 
fueled education and training programs delivered through schools, government agencies, 
and industry. However, access to these resources requires a base of developed literacy 
skills often lacking in the very populations they target. When scaled to meet the demand 
of the numbers of needy students, the costs of instructor-led solutions become 
impractical, inviting a search alternatives based on technology. 

Enormous gains in processor speed have combined with the development of 
powerful algorithms to deliver automated speech recognition (ASR) technologies 
effective enough to support computer-based language and literacy tutors. For example, a 
team from IBM Research re-purposed IBM’s dictation-oriented ASR by equipping it with 
an acoustic model appropriate for six year-olds, creating a usable, provably effective 
reading tutor [4]. 

Extending this work by using a web-based architecture (see Figure 5) to centralize 
ASR resources and instructional management will yield several advantages: 

• A centralized architecture enables matching users with the acoustic model that 
best recognizes their speech, enabling optimization for linguistically diverse 
groups 

• Background data collection supports the development of individualized acoustic 
models to optimize speech recognition for each learner 

• Centralized session management will enable learner to collaborate, exploiting 
validated approaches such as reciprocal teaching [5] 

• Workflow-governed collaborative content development strategies will quickly 
yield large amounts of sharable narrative and expository training materials. 
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Figure 4 



Figure 5 
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TRANSLATION 


While the diffusion of information and communication technologies promises 
universal access to people, resources, and services for learning, language barriers belie 
that promise for most of the world's population. For example, the predicted 280 million 
English-speakers to enjoy the Web in 2004 will be joined by as many as 680 million 
speakers of other languages, condemned to use it only fractionally [6]. 

The preponderance of English creates a linguistic equivalent of the digital divide. 
Sixty percent of the pages on the Web are composed in English, a language used by only 
29% of those who use the Web [7]. As the proportion of pages in English has shrunk 
from 64.55% in 1999 to 60.75% in 2001, this <4% decline implies increases in other 
languages (e.g., German, 23%; Chinese, 72%; Arabic, 750%). Still, these stunning 
proportional increases are minuscule in absolute terms. 

This linguistic digital divide is present in our local schools too. Language 
differences bedevil learning in our linguistically heterogeneous schools. More than 325 
languages are used at home in the U.S [8], New York City, for example, enrolls 15% of 
all its students in English language learning classes; in Los Angeles, 45% [9]. Although 
eighty percent of these students use Spanish in their homes, other languages shoulder 
their way to significant frequencies. For example, of the top three foreign languages in 
New York City students' homes, Spanish leads with 82%, Chinese follows with 12%, and 
Russian fills the remainder with about 6% [9]. 

To combat this linguistic Balkanization, teams at IBM Research pursue both 
grammar-based and statistical approaches to language translation. Spurning a 
dichotomous tradition of using only one technique (cf: [10]), they build upon the 
complementary approaches of each. The expense of hand-building frame dictionaries 
needed for grammatically motivated approaches limits them to translation efforts 
operating between pairs of languages deemed significant for global commerce. In 
contrast, statistical approaches can feed off corpora of existing translations to bootstrap a 
translation system. 

IBM Research teams have yoked the power of this hybrid approach translation 
technologies to its pioneering automatic speech recognition work to produce the Speech- 
to-Speech Translator [11], Even though it links the two uncertain channels of speech 
recognition and language translation, it combines with visual and extralinguistic cues 
available to those engaged in conversations common to classroom learning. 
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SUPPORTING REFLECTION VIA E-PORTFOLIOS 


Schools are curious institutions when contrasted with learning through 
apprenticeship: Schools separate students from the places of work and activity where 
knowledge is being demonstrated and developed, whereas apprenticeship learning takes 
place in situ within the activity context. Ethnographic and historical studies of 
apprenticeship list impressive advantages for learning, especially for fields where much 
of the knowledge is process oriented and not highly codified. In apprenticeships learning 
occurs in context of the tasks being performed; the learner observes expert practice and 
can imitate it; and the teachers are experts in the domain. How can the distance between 
the academy and actual contexts of use be reduced? One approach is to help experts 
document their practice in portfolios. 

Portfolios can serve as the basis for reflection, collaborative discussions, and the 
refinement of tacit knowledge into explicit principles and techniques. Over time these 
portfolios can help experts evolve a more formal understanding and codification of their 
knowledge. These are the goals, in part, of the IBM Reinventing Education Project 
which is seeking to improve teacher preparation. The functional component diagram 
below (Figure 7) shows how such a system might be organized. The rounded rectangles 
describe processes (e.g.. Organizer, Evaluator), while the rectangles describe data (e.g., 
portfolios, standards). A time-slice through the system might work as follows: A teacher 
uses the organizer to create a portfolio. This process involves importing artifacts (e.g., 
lesson plans and student work), attaching meta-data (e.g., date of origin), and adding 
reflections and analysis that link artifacts with relevant standards or their professional 
development plan. Beyond their personal reflection, a teacher can engage the peers and 
mentors via on-line meetings and interactions. When the portfolio reaches a level of 
completion it can be submitted for evaluation which invokes a workflow process where it 
is reviewed. These reviews might lead to changes. Eventually, the portfolio can be 
published for a broader audience to access. In these ways, hard won “in the trenches” 
knowledge can be captured, refined, and built-on, thus building a scholarship based on 
practice and integrated with theory. 
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Figure 6 



Figure 7 
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RELEVANT INFORMATION ON DEMAND 


Asking questions is at the heart of much learning. Important advances have been 
made in systems that automatically provide useful answers to certain types of questions. 
These systems work by analyzing a “document base” or scores of documents that contain 
the answers the system will provide. For example, a question and answer system (Q&A) 
might use an encyclopedia as its base input. When a student asks a question, the system 
retrieves the part of the encyclopedia best suited to answering that question. Currently, 
systems of this sort work best for “shallow” questions like Who, What, When, and Where, 
but less well for “deeper” questions like Why (Not), What If and How Did It Happen 
[12]. The Figure below shows the architecture of a system that uses predictive annotation 
(for a more complete description of the systems illustrated in the figure please see [13]). 
This means that as parts of the document base are analyzed tagged entries are created 
organized by the type of answers that they might provide. For example, “Benjamin 
Franklin invented the Post Office”, would create a “Who” entry if “Who” were asked 
about “Post Office”. Similarly, when a question is asked, tagged entries are created from 
it. Answers are found by matching the entries generated by the question to the ones 
generated from the document base of potential answers. 



Question 

.Templates 


Who -> PERSONS 

Where -> PLACES 

How much .. cost -> MONEYS 

How long/tall/.. .-> LENGTHS 


MONEYS 



LENGTHS 



QA- Patterns 


Index 


uer D- 
How tall is the 
Matterhorn? 


Template-Matching 
Lemmatization 
Stop-word Removal 


LENGTHS Matterhorn 


GuruQA 

Search 


C5 


nswer 


Answer 

Selection 




Hit-List 


Question- 

Answering 

http://icecream.watson.ibm.com/qa_eb.html 


A landmark climb in the growth of the sport was the 
spectacular first ascent of the Matterhorn (14,692 feet) on July 
14, 1865, by a party led by an English artist, Edward Whymper. 
mountaineering 

Edward Whymper, English mountaineer and artist who was 
associated with the exploration of the Alps and was the first 
man to climb the Matterhorn (14,691 feet [4,478 metres]). 
Whymper, Edward 


Figure 8 


402 


“JUST-IN-TIME” TRAINING FOR RETAIL EMPLOYEES 


Performance support is best if it can be provided right where and when it’s 
needed, rather than segregating it to class time instruction. This need is often keen in 
retail settings, since workers are expected to learn the systems quickly with as little time 
devoted to learning as possible. Exacerbating difficulties include high employee 
turnover, poor motivational structures, timed training time, lack of space in the stores, 
and short shelf life (with the consequent need to learn about the new stock). Beyond new 
employee training, all employees periodically require additional training in areas such as 
safety, security, store layout, company procedures, product information, and employee 
relations. 

Retail requires a multi-tier approach where different technologies are used to 
address training needs with differing characteristics. Training can be informational, 
interactive, collaborative, or co-located.. 

Perhaps the introduction of “just-in-time” information to be the most important 
new development in retail training that has been missing from prior approaches. 
However, generating materials dynamically poses a considerable technical problem. 
Templates can be created to tailor information to various types of learners (senior, out of- 
college) and to take into account regional considerations. Further, information can be 
transcoded to accomodate wireless handheld devices with small screens or web kiosks on 
the floor of the retail outlet. 


| IBM Institute for Advanced Learning 



e-leaming for retail employees 

Technology 

• Handheld 

• Very small footprint 

• Mobile (clipped to a belt) 

• Wireless device with 802.1 1 

• Easy log-in 

• Profile-based 

• Text, voice, audio, visual capable 

• Not easy to steal or lose 

• Not easily damaged 

• Easily repaired, maintained and supported 

Content 

• Access to corporate learning portal 

• Real-time content 

• Product and zone information 

• Small learning objects 

Business 

• Must not interfere with customer interaction 
^^£ostpoint^$1000^ 


Figure 9 
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LEARNING OBJECTS FRAMEWORK 


In recent years a substantial standards effort has grown up around creating 
learning objects that can be broadly interoperable and reusable. This effort could prove a 
great boon for publishers, corporations, and individual instructors and teachers. However, 
a key problem is achieving the reuse or repurposing of existing media objects (i.e., 
content) that were not originally designed for delivery as training. For example, such 
content includes how-to manuals, textbooks, presentations, or web sites. 

The Learning Objects Framework project is developing new models for 
generating on-demand web-based learning experiences from interchangeable modular 
learning objects. The framework consists of software components for disassembling, 
filtering, and automatically processing content to create learning objects and for 
reassembling learning objects into dynamic courseware. Using this framework, the group 
has developed a learning environment where users discover and assemble modular 
learning objects into short, personalized web-based training courses as needed. A key 
problem is how to maintain coherence among diverse assembled materials. Through this 
work the project has participated in the development of e-learning standards. 
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SEGMENTATION OF LEARNING VIDEOS 
BASED ON NARRATIVE STRUCTURE 


Instructional videos can provide a rich source of learning objects, if the relevant 
sections could be automatically retrieved. Towards this goal, IBM Research is 
developing technologies and middleware tools for standards-based e-learning content 
management to support shareable and searchable learning object repositories with rich 
media. This effort includes automatic analysis of audio and video for automated 
extraction of metadata for content tagging. 

The video segmentation technology seeks to automatically analyze and segment 
instructional media using high-level semantics-based video and audio features, thereby 
enabling effective search and browsing of instructional videos [14]. The analysis 
automatically detects and separately considers the video’s dialogue, narrative, raw 
footage, and text. The detection relies on common rhetorical conventions. For example, 
directed narrative is detected by looking for video that is dominated by the speaker’s 
face, whereas in discussion sections the audio track is dominated by the narrator’s voice, 
but the narrator does not appear. Once analyzed and segmented, the clips are formed into 
learning objects. Thus, user’s queries can be addressed with focused video clips that 
were automatically derived from instructional videos. 

The audio analysis is done using a configured IBM ViaVoice system, while 
custom algorithms have been developed to analyze the video. The system generates both 
MPEG-7 compliant metadata and Sharable Content Object Reference Model [15] 
(SCORM) compliant metadata suitable for loading into content management systems. 




Figure 1 1 
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COLLABORATION PATTERNS 


Most significant accomplishments are the result of distributed performance 
amongst many people. Yet, the sheer complexity of large organizations and 
collaborations make it difficult for us to comprehend and navigate the social networks of 
which we are part. Visualization systems that map these relationships offer profound 
advantages for better making use of our social networks, including the choice of whom to 
ask for help, planning organizational changes, managing collaborative work, and 
maintaining strong networks of colleagues [16]. In the figure below, a social network is 
shown based on the co-authorship of patents at a large retail products firm. The green 
nodes represent patents, while the yellow and blue nodes represent inventors. The blue 
nodes show a single master inventor and his co-authors. These same inventors are 
profiled, minus the master inventor and the patents, in the network inset on the right. 
Showing the social network without the master inventor or “hub node” highlights other 
inventors who are showing significant leadership and how the lab’s social network would 
look if the master inventor left. 




File Edit View Layout Graph Index Tools Help 



Figure 12 
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SUMMARY AND CONCLUSIONS 


Advanced technologies offer the promise to profoundly impact how we learn and 
what we need to know in order to accomplish a wide variety of tasks. In reviewing a 
suite of IBM technologies we have argued advanced technologies are having a profound 
impact on the context, means, and goals of learning. 

In particular, technologies have the ability to transform: 

1. Who learns : Access to learning can be opened up to a wider population by 
overcoming disabilities, geography, and language barriers, and providing more 
universal access. 

2. What needs to be learned : Technologies change what role people play in the 
accomplishment of tasks, thus difficult education problems may disappear entirely 
and the learner can be raised to a new level with increased cognitive abilities. 

3. When learning happens: Learning can be better integrated with work, so learning 
occurs when needed. In part, this is enabled by using devices of diverse sizes and 
abilities, ranging from wireless handhelds to large displays with gesture 
recognition. 

4. Where learning happens: The ability of computers to integrate information with 
our problem context, or to situate it, means that learning can occur on the street, at 
home, and, in general, integrated with our interactions in the world. In short, 
learning becomes ubiquitous. 

5. How learning happens: Translation and language teaching software can help 
support people communicate using diverse languages. Further, reflective 
practioners can be provided with more supportive environments for learning from 
their practice. Finally, improved interaction with large and complex social 
systems can be enabled by social network analysis systems. 

In short, technology can better support anytime and anywhere access to situated 
knowledge at an increased cognitive depth. These far reaching improvements underline 
the importance of creatively designing technologies to aid our learning. 
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